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Executive Summary 

The No Child Left Behind Act of 2001 (PL 107-110) established the Reading First Program 
(Title I, Part B, Subpart 1), a major federal initiative designed to help ensure that ah children can 
read at or above grade level by the end of third grade. Reading First (RF) is predicated on 
scientifically researched findings that high-quality reading instruction in the primary grades 
significantly reduces the number of students who experience reading difficulties in later years. 

The legislation also mandates the U.S. Department of Education (ED) to contract with an outside 
entity to evaluate the program’s implementation (Section 1205). During the 2004-05 and 2006- 
07 school years this evaluation collected survey data from principals, teachers and reading 
coaches in nationally representative samples of RF schools and non-RF Title I schoolwide 
project (SWP) schools (e.g., schools in which at least 40 percent of the students are eligible for 
free or reduced price lunches and have elected to implement programs in which all children are 
eligible for Title I programs and services). Across ah respondents (principals, reading coaches 
and teachers), the overall response rate was over 90 percent for both waves of survey data 
collection. Additionally, extant data on school-level scores on third- and fourth-grade state 
reading assessments were analyzed to examine reading achievement trends in these two groups 
of schools. 

Major Findings 

• Reading programs implemented in RF schools differ from those in non-RF Title I schools 
in several ways: RF schools devote more time to reading instruction in K-3 classrooms, 
and are more likely to: a) have reading coaches who assist teachers in implementing their 
reading programs; b) use reading materials aligned with scientifically based reading 
research; c) use assessments to guide instruction; d) place struggling readers into 
intervention services; and e) have their teachers participate in reading-related professional 
development. 

• Non-RF Title I schools increasingly report activities aligned with the principles of 
Reading First, including a) providing assistance to struggling readers, b) teacher 
knowledge and use of materials and strategies aligned with scientifically based reading 
research, and c) staff participation in professional development in the five dimensions of 
reading instruction (phonemic awareness, phonics, vocabulary, fluency and 
comprehension) . 

• Based on analyses of states’ reading assessment scores, there is limited but statistically 
significant evidence that successive cohorts of third- and fourth-grade students in RF 
schools improved their reading performance over time more quickly than did their 
counterparts in non-RF Title I schools. There is a positive and statistically significant 
relationship between only one of four measures of RF and non-RF Title I schools’ 
implementation of RF-aligned activities, as measured through surveys, and their levels of 
third-grade reading achievement. 
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Key Provisions of the Reading First Program 



The Reading First program’s overarching goal is to improve the quality of reading instruction — 
and thereby improve the reading skills and achievement of children in kindergarten through third 
grade — ^by implementing the use of research-based instruction and materials. Reading First aims 
to increase both the availability and quality of professional development for all K-3 teachers, 
including special education teachers, so that they have the necessary skills to teach research- 
based reading programs effectively An important provision of the RF legislation is that 
professional development be available to staff in all schools, not only those with RF funding. 
Specifically, K-3 teachers are eligible to participate in professional development paid for by 
district RF funds, and K-12 Special Education teachers are eligible to participate in professional 
development paid for by state RF funds. 

Reading programs and instruction methods should incorporate the five essential elements of 
effective primary grade reading instruction, specified in the legislation: 1) phonemic awareness; 
2) phonics; 3) vocabulary development; 4) reading fluency, including oral reading skills; and 5) 
reading comprehension strategies. Reading First also emphasizes the use of assessments, both to 
monitor progress and to identify and address students’ reading problems early, by helping 
classroom teachers to screen for, identify, and overcome barriers to students’ ability to read at 
grade level by the end of third grade. 

All 50 states and jurisdictions except Puerto Rico* have been awarded Reading First grants. To 
date (December, 2007), states have awarded subgrants to approximately 1,800 local school 
districts and, in turn, these districts have provided funds to approximately 5,100 schools 
nationwide. Because grants to states were awarded over an extended time period, and states 
differed in the amount of time they allotted to their competitive subgrant processes, districts and 
schools are at various stages of implementing their Reading First programs. 

Overview of the Evaluation 

To address the legislative requirement mandating an evaluation of Reading First, the U.S. 
Department of Education contracted with Abt Associates in October 2003 to design and conduct 
the Reading First Implementation Evaluation.^ The evaluation addresses the following 
questions: 

1 . How is the Reading First program implemented in districts and schools? 



' All states, the District of Columbia, Puerto Rico, the schools of the Bureau of Indian Education, and the 
outlying areas were eligible to apply for RF grants. Guam and Northern Mariana Islands received grants 
through the consolidated grants to insular areas. 

^ Although private schools students and teachers are eligible to receive equitable services under the Reading First 
Program, this evaluation includes only public schools and does not address RF programs implemented in private 
schools. 
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2. How does reading instruction differ between Reading First schools and non-RF Title I 
schools? 

3. How does reading instruction differ between Reading First schools and non-RF Title I 
schools as RF schools’ implementation efforts mature over time? 

4. Does student achievement improve over time more quickly in schools with Reading 
First funds than in non-RF Title I schools not receiving RF funds? 

5. Is there any relationship between how schools implement Reading First-aligned 
practices and changes in reading achievement? 

To date, this five-year study has produced The Reading First Implementation Evaluation: 

Interim Report which presented findings that addressed questions 1 and 2. This final report 
addresses questions 3-5. The data sources used in the current report include: 

• Surveys completed in spring 2005 and 2007 by K-3 teachers, principals, and reading 
coaches in nationally representative samples of Reading First schools and non-RF Title I 
schools; and 

• State and national databases on school-level reading scores on state assessments across 
all school districts nationwide. 

Analysis of Survey Data 

The study included two nationally representative samples of RF schools. The first is a group of 
mature schools'^ that had been implementing RF activities for one or more years, and the other is 
a group of newly funded schools that were beginning implementation during the 2004-05 school 
year. In 2007, only the latter sample of RF schools was surveyed again. Collecting survey 
information at two time periods allowed us to address question 3 about how the reading 
programs in RF schools mature over time and RF grants had ended in many of the mature 
schools by 2006-07. 

The non-RF Title I school sample was constructed purposefully to provide a context for 
understanding how reading programs in a sample of Reading First schools differ from those in 



^ In 2004-05, staff from 1,633 schools completed surveys (1,092 RF and 541 non-RF Title I). In 2006-07, staff 
from 1,018 schools completed surveys (579 RF and 439 non-RF Title I schools). 

For the purpose of recruiting these two types of RF schools, schools were designated as new or mature based on 
the Reading First program guidelines to states for their annual performance reports; “. . .for schools receiving 
grants between July 1 and December 3 1 of any reporting period, the current school year will be considered in 
the first year of implementation.” Therefore, schools awarded subgrants before Dec. 31, 2003, were designated 
as mature, because data collection occurred in spring 2005, when those schools were in at least their second 
year of implementation. All other RF schools were classified as newly funded as of the beginning of the first 
data collection in 2005. 
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schools serving similar populations of students.^ This sample includes only Title 1 sehoolwide 
project (SWP) schools.^ This sample of sehools was surveyed at both time points. The RF and 
non-RF Title 1 schools are similar in staff experience, attendanee rates, mobility, and stability of 
enrollment. RF sehools are larger than the non-RF Title I sehools, on average, and have larger 
proportions of K-3 students reading below grade level. 

This final report uses the survey data first to deseribe and eompare the reading programs in RF 
and non-RF Title I sehools based on the 2006-07 survey data eolleetion. These data about the 
status of sehools’ reading programs represent the primary focus of this final report. Second, the 
report deseribes ehanges in sehools’ reading programs from 2004-05 to 2006-07, refereneing 
surveys fielded in 2004-05. Taken together, these findings address question 3: How does 
reading instruetion differ between Reading First schools and non-RF Title I schools as RF 
schools’ implementation efforts mature over time? 

Analysis of Student Reading Achievement 

This report also ineludes findings from two analyses of reading achievement trends. One 
examines third- and fourth-grade reading performanee on state reading assessments in order to 
address evaluation question 4: Does student aehievement improve over time more quiekly in 
schools with Reading First funds than in non-RF Title I schools not receiving RF funds? The 
seeond analysis examines the relationship between teachers’ RF-aligned instruetional activities 
and student reading aehievement, using 2005 survey data and states’ 2005 reading assessment 
scores; these findings address evaluation question 5: Is there any relationship between how 
schools implement Reading First-aligned practices and changes in reading achievement? 

Reading Performance on Third- and Fourth-Grade State Assessments 

Analyses were eonducted using annual third- (24 states) and fourth-grade (17 states) test seores 
on state reading assessments from 12,362 sehools (3,000 RF and 9,362 non-RF Title 1). 
Differenee-in-differenee models were fit to the data to estimate whether ehanges in reading 
performanee from pre- to post-RF implementation were larger in RF sehools than in non-RF 
Title 1 sehools. These analyses required specifying, for each school, which school years and 
their eorresponding test seores preceded the implementation of RF, and whieh sehool years and 
scores represent post-RF implementation. Because no data provide this information directly, we 
used four different methods to estimate schools’ pre- and post-RF implementation years: 1) 
initial state award date; 2) district award date; 3) an adjusted distriet award date to aecount for 

^ The most rigorous design option available for this evaluation would have been to identify a group of non-RF 
sehools matched to RF schools on key demographic and achievement characteristics to minimize differences 
between RF and non-RF schools and thereby attempt to approximate a random assignment experiment. 
However, because RF schools, by definition, are among the lowest-performing schools in their respective 
districts matched comparison schools could well have included better-performing schools. Also, RF schools 
could likely differ Ifom similar non- funded schools because they had to demonstrate motivation in order to 
participate, which could have influenced any observed instructional differences between RF and comparison 
schools. 

^ Title 1 schoolwide project schools are schools in which at least 40 percent of the students are eligible for fi'ee or 
reduced price lunches and have elected to implement programs where all children are eligible for Title 1 
programs and services. 
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the time between district awards and schools’ receipt of funds; and 4) school award dates based 
on RF principals’ survey responses.’ 

For each state, separate analyses were conducted using each of the methods described above, 
depending on data availability (at least one year of pre- and one year of post-RF implementation 
scores). In addition, we computed a weighted average effect size across states of the pre-RF to 
post-RF implementation differences in reading scores in RF and non-RF Title I schools by 
averaging weighted effect sizes calculated from the difference-in-difference estimate, for each of 
four methods used to define pre- and post-RF implementation years across both grades. 

Relationship of Reading Achievement Analysis to the Reading First Impact Study 

The findings summarized below describe national trends in reading achievement in RF and non- 
RF Title I schools nationally; these findings do not, however, represent a measure of the impact 

O 

of Reading First. The Reading First Impact Study, conducted by the U.S. Department of 
Education’s Institute of Education Sciences, is a five-year evaluation that uses a rigorous design 
(regression discontinuity design) to compare two groups of schools, those that did and did not 
receive Reading First funding to measure its impact on student reading. The study is conducted 
in over 240 elementary schools in 18 sites and 13 states, and is collecting information on 
students and classrooms in grades 1-3 over a three-year period. 

RF-Aligned Activities and Student Reading Achievement 

We used two data sources to investigate the relationship between student reading achievement 
and schools’ implementation of RF activities: 1) third-grade reading scores on 2005 state 
assessments; and 2) composite variables based on 2005 survey questions about classroom 
reading instruction, help for struggling readers, professional development, and use of 
assessments. Because tests and metrics vary across states, each state’s reading scores were 
standardized by designating schools as either high- or low-performing relative to all other RF 
and Title I SWP schools in that state. ^ Schools were then selected from the highest and lowest 
quartiles; the analytic sample included 390 schools from 21 states. 

To explore the relationship between teachers’ implementation of RF-aligned activities and 
schools’ performance on states’ third- grade reading assessments, we fit a logistic regression 
model using the four composites to predict the probability that a school scored in the top quartile 
(of all RF and non-RF Title I SWP schools) on their state’s third-grade reading assessment. The 



’ Descriptions of the four methods for assigning start dates and pre- and post-RF implementation years are 
presented in Chapter 8 and in Appendix C. 

* Gamse, B.C., Bloom, H.S., Kemple, J.J., Jacob, R.T., (2008). Reading First Impact Study: Interim Report 
(NCEE 2008-4016). Washington, D.C.: National Center for Education Evaluation and Regional Assistance, 
Institute of Education Sciences, U.S. Department of Education. 

^ For each state, we selected schools in the lowest quartile (at or below the 25th percentile) and the highest 
quartile (at or above the 75th percentile) on the state’s 2005 third-grade reading assessment relative to all RF 
and Title I SWP schools. 
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model also included indicator variables for each state, to account for RF status and for state level 
variation in each school’s probability of being in the top quartile on the reading assessment. 

The findings from the analyses of student reading achievement should be interpreted with 
considerable caution. Although the RF and non-RF Title I schools are similar demographically 
on those characteristics measured in the study, schools were not randomly assigned to receive RF 
funding, and therefore there could have been preexisting differences between the two groups of 
schools on unobserved characteristics. We cannot assume that the two groups of schools are 
equivalent but for the fact that one group received RF funds while the other did not. This means 
that the findings cannot support causal inferences that attribute observed differences in student 
reading achievement between RF and non-RF Title I schools to the Reading First program.'*^ 

Key Findings 

In this section, we summarize the three key findings presented in this report: 1) differences in 
reading programs between RF and non-RF Title I schools; 2) the increasing prevalence of 
Reading First activities in non-RF Title I schools; and 3) student reading achievement trends in 
RF and non-RF Title I schools. In general we find reading practices in RF schools and non-RF 
schools are similar in many ways, and have changed similarly over time in ways that are 
consistent with RF principles. We also find across a variety of indicators, that reading 
instructional time, professional development, use of reading resources and supports is more 
widely available or extensively utilized in RF schools. 

Differences in the Reading Programs between RF and Non-RF Titie i Schoois 

Reading First schools spent more time on reading instruction, use reading materials and 
instructional strategies aligned with scientifically based reading research (SBRR), and are 
more likely to provide additional supports and interventions for struggling readers than 
non-RF Title I schools. 

Instructional Time. RF schools dedicated more time to reading instruction in their K-3 
classrooms than did non-RF Title I schools. Most RF and non-RF Title I schools scheduled 
designated reading blocks for their K-3 classrooms, although these blocks were longer in RF 
schools; according to principals and reading coaches, 98 percent of RF schools and 77 percent of 
non-RF Title I schools scheduled reading blocks for at least 90 minutes. K-3 teachers in RF 
schools reported spending an average of 103 minutes per day on reading activities, compared to 
81 minutes reported by teachers in non-RF Title I schools. This translates to approximately 110 
additional minutes per week of reading instruction for K-3 students in RF schools. 

Instructional Materials. After making substantial changes to their reading programs during 
their first year of implementation in 2004-05, mature RF schools were less likely to make many 
additional changes in 2006-07 than were non-RF Title I schools. Only 3 percent of RF schools 



Additional limitations to the analysis of student reading achievement are presented in Chapter 8. 
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adopted a new core reading program in 2006-07, compared to 39 percent in 2004-05, and only 
14 percent added new materials for English Language Learners (ELLs), compared to 43 percent 
in the earlier period. However, some RF schools continued to add new intervention programs for 
struggling readers (40 percent) and add new supplementary materials to their reading programs 
(42 percent) in 2006-07. 

SBRR Alignment. Survey responses indicate that reading instruction in RF schools was more 
likely to be aligned with SBRR than in non-RF Title I schools. Based on surveys of reading 
coaches, RF schools were more likely to report that their K-3 teachers are knowledgeable about 
SBRR (79 percent vs. 58 percent), that core reading programs in RF schools are more likely to 
be aligned with SBRR than are the programs in non-RF Title I schools (93 percent vs. 76 
percent), and that reading intervention materials in their schools are aligned with SBRR (94 
percent vs. 79 percent) than were their counterparts in non-RF Title I schools. In addition, more 
teachers in RF schools rated SBRR-aligned practices central to their instruction than did teachers 
in non-RF Title I schools (85 percent vs. 75 percent)." 

Assistance to Struggling Readers. In 2006-07, RF teachers were more likely than teachers in 
non-RF Title I schools to report placing struggling readers into intervention services (80 percent 
vs. 63 percent), and to report using diagnostic assessments to determine struggling readers’ core 
deficits (84 percent vs. 67 percent). 

Assessment. While assessment plays an important role in reading programs of both RF and non- 
RF Title I schools, there is evidence that assessment is a more integral element of reading 
instruction in RF schools than in non-RF Title I schools. Teachers in RF schools were more 
likely than teachers in non-RF Title I schools to rate the use of assessment results as central to 
their instruction for such purposes as grouping students (91 percent vs. 78 percent), identifying 
students in need of interventions (83 percent vs. 69 percent), and measuring student progress (88 
percent vs. 80 percent). 

In 2006-07, RF teachers were significantly more likely than teachers in non-RF Title I schools to 
identify standardized tests as the assessment used most often and to administer such tests more 
often than teachers in non-RF Title I schools. Informal assessments were rarely characterized as 
the most frequently used assessments, yet they were more likely to be identified by teachers in 
non-RF Title I schools than teachers in RF schools. 

Reading Coaches. RF schools were significantly more likely to have a reading coach than were 
non-RF Title I schools (99 percent vs. 57 percent). Reading coaches in RF schools reported 
spending larger proportions of their time as in the role of reading coach than did coaches in non- 
RF Title I schools; specifically, 75 percent of RF coaches and 19 percent of coaches in non-RF 
Title I schools reported that they spend all of their time in this role. 



" This finding is based on a composite measure of SBRR that reflects the percentage of instructional activities 
focused on the five dimensions of reading that teachers rated as central to their instruction. (The specific 
activities included in the SBRR composite are listed in Appendix D). 
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Professional Development. In 2006-07, a greater percentage of RF teachers reported 
participating in reading-related professional development activities than did teachers in non-RF 
Title I schools (90 percent vs. 73 percent). On average, teachers in RF schools reported 
participating in more than twice the number of workshops (4.9 vs. 2.3) and spending twice the 
number of hours in reading-related professional development activities (31.3 hours vs. 15.7 
hours) than did teachers in non-RF Title I schools. However, the average number of workshops 
and hours teachers participated in reading-related professional development declined for teachers 
in both RF and non-RF Title I schools from 2004-05 to 2006-07. For example, the number 
workshops declined from 7.0 to 4.9 for RF teachers and from 3.3 to 2.3 for teachers in non-RF 
Title I schools. 

Further, in 2006-07 teachers in RF schools were more likely than teachers in non-RF Title I 
schools to report participating in professional development on the five dimensions of reading 
including comprehension (88 percent vs. 74 percent), phonics (88 percent vs. 64 percent), 
phonemic awareness (87 percent vs. 62 percent), vocabulary (82 percent vs. 60 percent) and 
fluency (91 percent vs. 74 percent). This same pattern of participation in professional 
development activities on the five dimensions of reading was also evident in 2004-05. 

Direct Classroom Support. In addition to formal professional development, RF teachers were 
more likely to report receiving ongoing direct support and feedback for their classroom reading 
instruction than were teachers in non-RF Title I schools. A greater proportion of teachers in RF 
schools than in non-RF Title I schools reported receiving classroom support on: interpretation of 
assessment data (91 percent vs. 70 percent); assistance from a reading coach or specialist in 
diagnosing individual student needs (72 percent vs. 48 percent); and intervention service help for 
individual students (73 percent vs. 52 percent). 

Reading First Activities in Non-RF Titie i Schoois 

Although Reading First and non-RF Title I schools’ reading programs continue to differ in 
various ways, RF-aligned activities are increasingly occurring in non-RF Title I schools. The 
areas of alignment include: a) changes in instructional materials; b) additional practice 
opportunities and direct instruction to help struggling readers; c) knowledge and use of materials 
and strategies aligned with scientifically based reading research; and d) participation in 
professional development in the five dimensions of reading instruction (phonemic awareness, 
phonics, vocabulary, fluency and comprehension). 

Instructional Materials. Substantial numbers of RF and non-RF Title I schools reported 
making changes to the instructional materials used in their reading programs. In 2006-07, 40 
percent or more of RF and non-RF Title I schools reported that they had added new intervention 
programs for struggling readers during the 2006-07 school year. Further, approximately one- 
third of both RF non-RF Title I schools added new supplementary materials to their reading 
programs in 2004-05 and then again in 2006-07 (33 percent and 38 percent, respectively). 
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Four of the five reading programs cited most frequently by RF schools were also among the five 
most popular programs in the non-RF Title I schools. At least 10 percent of both RF and non-RF 
Title I schools reported using Harcourt Trophies, McGraw-Hill Open Court, Scott Foresman 
Reading and Houghton Mifflin Reading. 

SBRR Alignment. The proportion of reading coaches in non-RF Title I schools rating their K-3 
teachers as knowledgeable about scientifically based reading instruction increased from 48 
percent in 2004-05 to 58 percent in 2006-07 which is about the same level as reading coaches in 
RF schools reported in 2004-05 (57 percent). In 2006-07 reading coaches in 79 percent of RF 
schools rated their teachers as knowledgeable about SBRR. 

Assistance to Struggling Readers. Materials and activities specifically aimed at helping 
struggling readers were available in most non-RF Title I schools (85 percent) and RF schools (91 
percent). Reading coach and principal surveys indicated that teachers provide struggling readers 
with additional direct instruction and practice opportunities in more than 95 percent of RF and 
non-RF Title I schools (with reading coaches); in about 90 percent of both RF and non-RF Title I 
schools, trained aides or volunteers work with struggling readers during class. 

From 2004-05 to 2006-07, the proportion of teachers in both RF non-RF Title I schools who 
reported that they had placed struggling readers into intervention programs in the last month 
increased modestly, from 73 to 80 percent in RF schools, and from 56 to 63 percent in non-RF 
Title I schools. As was the case for teachers in RF schools, the percent of teachers in non-RF 
Title I schools who reported that time is set aside to coordinate with ELL staff about the reading 
instruction provided to struggling readers increased from 40 percent in 2004-05 to 70 percent in 
2006-07. 

Professional Development. In 2006-07, many K-3 teachers in non-RF Title I schools reported 
that they participated in professional development on the five dimensions of reading: reading 
comprehension (74 percent); phonics (64 percent); phonemic awareness (62 percent); and 
fluency (74 percent). Also, the percent of teachers in non-RF Title I schools who reported they 
participated in professional development on vocabulary increased from 5 1 percent in 2004-05 to 
60 percent in 2006-07. 

Student Reading Achievement Trends in RF and Non-RF Titie i Schoois 

Based on analyses of states’ reading assessment scores, there is limited but statistically 
significant evidence that successive cohorts of third- and fourth-grade students in RF schools 
improved their reading performance over time more quickly than did their counterparts in non- 
RF Title I SWP schools. There is a positive and statistically significant relationship between 
only one of four measures of RF and non-RF Title I schools’ implementation of RF-aligned 
activities, as measured through surveys, and their levels of third-grade reading achievement. 

Third-Grade Reading Performance 

On third-grade state reading assessments, average effect sizes across 24 states indicate that RF 
schools gained between 2 and 3 percentage points more, on average, from pre- to post-RF 
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implementation than non-RF Title I schools on the proportion of students meeting standards on 
states’ third- grade reading assessments, a statistically significant yet small difference (p < 

.001). In 12 of 24 states, the improvement in third-grade reading scores among RF schools was 
statistically significantly larger than in non-RF Title I schools for at least one of the four methods 
used to define pre- and post-RF implementation years. In the other 12 states, there were no 
statistically significant differences between the two groups of schools. 

Fourth-Grade Reading Performance 

On fourth- grade state reading assessments, average effect sizes across 17 states indicate that RF 
schools gained between 2 and 3 percentage points more, on average, from pre- to post-RF 
implementation than non-RF Title I schools on the proportion of students meeting standards on 
states’ fourth-grade reading assessments, a statistically significant yet small difference (p < 

.001). In six of 17 states, the improvement in fourth-grade reading scores among RF schools was 
statistically significantly larger than in non-RF Title I schools for at least one of methods 
(described earlier) used in the analysis. In 1 1 states there were no significant differences 
between the two types of schools. 

Reading Performance and RF-aligned Practices 

There is a positive and statistically significant relationship between only one of four measures of 
RF-aligned activities, as measured through surveys, and schools’ levels of third-grade reading 
achievement. The study team analyzed the relationship between schools’ third-grade reading 
scores on state assessments and four composite measures constructed from survey data that 
characterize teachers’ RF-aligned activities: classroom reading instruction; strategies to help 
struggling readers; participation in professional development; and uses of assessment to inform 
instruction. 

Only the composite measure of teachers’ use of activities for struggling readers was statistically 
significantly related to the probability that a school scored in the top quartile (relative to other RF 
and non-RF Title I schools) on its state’s third-grade reading assessment. For every increase of 
one standard deviation unit in the struggling readers implementation composite score, the 
probability of being in the top quartile increased by 15.6 percentage points, for the average 
school (p<.001).*"^ 



The results of the pooled analyses were statistically significant at the p < .001 level for each of four analytic 
methods which are described in more detail in Chapter 8 and Appendix C. 

This analysis includes both RF and non-RF Title I schools, since the implementation of activities aligned with 
Reading First can potentially occur in either type of school. 

The analysis of the relationship among RF-aligned activities and reading achievement required fitting a logistic 
regression model because the dependent variable is dichotomous-inclusion or exclusion from the highest 
quartile. The four composites were standardized and are best understood in terms of standard deviation units. 
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Summary 



These findings provide some evidence that after several years of implementation, Reading First 
continues to be implemented in schools and classrooms as intended by the legislation. States 
continue to provide appropriate supports, including professional development related to reading, 
and assistance in the selection and use of assessments to inform instruction. Reading First 
schools continue to provide instructional environments that support SBRR-based instruction. 
There is also evidence that the reading programs implemented by teachers in Reading First and 
non-RF Title I schools’ K-3 classrooms continue to differ on instructional time, resources, 
instructional planning and collaboration and use of assessments. 

Although reading programs in RF and non-RF Title I schools differ, it is important to note that 
RF-like activities are increasingly occurring in non-RF Title I schools, which report increasing 
prevalence of practices aligned with the principles of Reading First. In 2006-07, teachers in 
non-RF Title I schools rated a higher proportion of scientifically based teaching strategies and 
materials as central to their instruction than they did in 2004-05. Teachers in these schools 
reported that they also have increased their level of effort to help struggling readers through use 
of diagnostic assessments to identify struggling readers and by placing these students in 
intervention programs. Also, from 2004-05 to 2006-07, an increasing percentage of teachers in 
non-RF Title I schools reported needing additional professional development in using 
assessments to guide instruction and to better assist struggling readers. 

The primary goal of Reading First is to improve student reading achievement such that all 
students are reading at or above grade level by the end of third grade. There is limited evidence 
that third- and fourth-grade students in RF schools improved their reading performance over time 
more quickly than did their counterparts in non-RF Title I schools. The differences were 
statistically significant yet small in magnitude, and given the limitations described earlier (and in 
detail in Chapter 8) these differences should be considered with caution. 
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Chapter 1 : Introduction 



The Reading First Implementation Evaluation Study is designed to address the following 
questions: 

1 . How is the Reading First program implemented in districts and schools? 

2. How does reading instruction differ in RF schools and non-RF Title I schools? 

3. How does reading instruction differ in RF schools and non-RF Title I schools as RF 
schools’ implementation efforts mature over time? 

4. Does student achievement improve over time more quickly in schools with RF funds 
than in non-RF Title I schools not receiving RF funds? 

5. Is there any relationship between how schools implement RF-aligned practices and 
changes in reading achievement? 

The Reading First Implementation Evaluation: Interim Report presented findings that addressed 
questions 1 and 2 (U.S. Department of Education, July 2006). This final report addresses 
questions 3 through 5. In this chapter, we review the context for the study, including the 
background, a description of the scientifically based reading research that informed the 
legislation, the Reading First legislation itself, the study design, analytic approaches, and 
limitations. In addition, this chapter summarizes findings from earlier reports. Last, we present 
the organization for the remainder of the report. 

Background 

The ability to read and comprehend text well is at the heart of educational attainment and, as 
such, is central to all children’s elementary school success. Unfortunately, success in elementary 
school (and beyond) disproportionately eludes many minority and economically disadvantaged 
children. Large numbers of children in high-poverty schools are not developing the reading 
skills needed for success in school. Results from the 2007 National Assessment of Educational 
Progress (NAEP) Reading Assessment indicate that 50 percent of fourth-grade students eligible 
for free or reduced-price lunches read at a Below Basic level compared to only 21 percent of 
fourth-graders not eligible for the lunch program who perform at that level. Although 
disappointing, these findings represent an improvement over the 2000 NAEP results, when 62 
percent of students eligible for free or reduced price lunches scored at the Below Basic level 
(U.S. Department of Education, 2007). 



Performance at the Below Basic level means that fourth-grade students are not performing at the Basic level. 
“Fourth-grade students performing at the Basic level should demonstrate an understanding of the overall 
meaning of what they read. When reading text appropriate for fourth-graders, they should be able to make 
relatively obvious connections between the text and their own experiences and extend the ideas in the text by 
making simple inferences.” (National Center for Education Statistics, retrieved May 10, 2006, from 
http://nces.ed.gOv/nationsreportcard/reading/achieveall.asp#grade4). 
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Scientifically Based Reading Research 

The fact that substantial numbers of our nation’s primary grade students are not developing 
adequate reading skills occurs at a time when we have made considerable progress in 
understanding how to teach reading effectively in the early grades, particularly to children who 
are struggling academically. The National Research Council’s 1998 report, Preventing Reading 
Difficulties in Young Children, noted, “The majority of the reading problems faced by today’s 
adolescents and adults could have been avoided or resolved in the early years of childhood.” 

The report noted that for children to learn to read effectively they must “understand how sounds 
are represented alphabetically, have sufficient practice in reading to achieve fluency with 
different kinds of texts, the background knowledge and vocabulary to extract meaning from text, 
and the motivation to read for varied purposes.” The report concluded that many elements of 
effective reading instruction are already known, and that the provision of “excellent instruction is 
the best intervention for children who demonstrate problems learning to read.” 

Building on the council’s report, the National Reading Panel reviewed the scientific research in 
key areas of reading development, focusing on skills critical to the acquisition of beginning 
reading skills.*^ The report found strong evidence that direct, explicit instruction is helpful to 
primary grade children in the development of their reading skills, particularly in the areas of 
phonemic awareness, phonics, vocabulary, comprehension, and fluency. 

The Reading First Legislation 

The No Child Left Behind Act of 2001 (PL 107-110) established the Reading First Program 
(Title I, Part B, Subpart 1) to address the fact that large numbers of our nation’s students do not 
develop the reading skills necessary to be successful in school. Reading First is a major federal 
initiative that builds on years of scientific research in reading to ensure that all children can read 
at or above grade level by the end of third grade. The legislation requires the U.S. Department of 
Education to contract with an outside entity to conduct an evaluation of, among other things, the 
Reading First (RF) program’s implementation (Section 1205). 

Reading First is predicated on research findings that high-quality reading instruction in the 
primary grades significantly reduces the number of students who experience difficulties in later 
years. The program’s overarching goal is to improve the quality of reading instruction and 
thereby improve the reading skills and achievement of children in the primary grades. The RF 
program provides substantial resources at both the state and local levels: 1) to ensure that 
research-based reading programs and materials are used to teach students in kindergarten through 
third grade; 2) to increase access to and quality of professional development of all teachers who 
teach K-3 students, including special education teachers, to ensure that they have the skills 
necessary to teach these reading programs effectively; and 3) to help prepare classroom teachers 



The National Reading Panel (NRP) was formed under the joint auspices of the National Institute of Child 
Health and Human Development and the U.S Department of Education to “assess the status of research-based 
knowledge, including the effectiveness of various approaches to teaching children to read” (Report of the 
National Reading Panel: Teaching Children to Read, 2000, page 1-1.) The findings from the NRP also were 
instrumental in the development of the Reading First program (part of the No Child Left Behind Act), the current 
administration’s comprehensive effort to improve early reading instruction and student reading achievement. 
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to screen, identify, and overcome barriers to students’ ability to read on grade level by the end of 
third grade. More specifically, the programs and the professional development provided to 
school staff must use reading instructional methods and materials that incorporate the five 
essential elements of effective primary-grade reading instruction, as specified in the legislation: 

1) phonemic awareness; 2) phonics; 3) vocabulary development; 4) reading fluency, including 
oral reading skills; and 5) reading comprehension strategies. 

The legislation was informed by prior research, and the guidance translated the legislation and 
prior research into more specific directions for states and their funded districts to follow. A 
central requirement of the RF legislation is that all instructional materials, activities and 
programs should be based on scientifically based reading research (SBRR) that provides 
evidence of effectiveness in teaching children to read. SBRR is defined in the legislation as: 

“. . .research that applies rigorous, systematic and objective procedures to obtain valid 

knowledge relevant to reading development, reading instruction, and reading difficulties. 

This includes research that: 

1 . Employs systematic, empirical methods that draw on observation or experiment; 

2. Involves rigorous data analyses that are adequate to test the stated hypotheses and 
justify the general conclusions drawn; 

3. Relies on measurements or observational methods that provide valid data across 
evaluators and observers and across multiple measurements and observations; and 

4. Has been accepted by a peer-reviewed journal or approved by a panel of independent 
experts through a comparably rigorous, objective and scientific review (Section 
1208).” 

In April 2002, the U.S. Department of Education invited state education agencies to apply for 
Reading First grants. State applications submitted to the U.S. Department of Education went 
through an expert review process that resulted in six-year awards dependent on demonstration of 
progress and congressional appropriations. States, in turn, awarded subgrants to local school 
districts based on a competitive process. All states and jurisdictions except Puerto Rico have 
been awarded Reading First grants. To date (December, 2007), states have awarded subgrants 
to approximately 1,800 local school districts and 5,100 schools nationwide.'^ 

There are two important aspects of program implementation that influence the findings reported 
here. First, the amount of time between states’ receipt of Reading First funding and districts’ and 
schools’ onset of Reading First program implementation varied considerably. Specifically, state- 
level grants were awarded between June 2002 and September 2003. States also varied in the 
length of their respective competitive processes, from several months to nearly a year. 



Guam and Northern Mariana Islands received grants through consolidated grants to the insular areas. 

The figures are based on current information listed on ED’s Reading First Awards Database (December, 2007, 
fromwww.sedl.org/readingfirst/reports-awards.html). 
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Additionally, states awarded subgrants of differing duration; some states funded schools for two 
or three years, others for up to six years. These factors resulted in district and school 
implementation of Reading First at different points in time. The study sample was constructed to 
take that variation into account by sampling from schools that (at that time) had varying amounts 
of implementation experience. 

Evaluations of Reading First 

The Implementation Evaluation is one of six complementary studies designed to gather 
information about Reading First at the federal level. In addition to the Implementation 
Evaluation, the U.S. Department of Education’s Policy and Program Studies Service (PPSS) 
conducted the Analyses of State Reading Standards and Assessments', that study evaluated the 
alignment of state reading content standards for students in grades K-3 with the five key 
elements of reading instruction by analyzing: 1) the reading content standards of a random 
sample of 20 states; and 2) the role of state assessments in measuring Reading First outcomes as 
presented in state Reading First applications. PPSS also is conducting the study of Reading First 
and Special Education Participation Rates Study, which uses an interrupted time-series design to: 
1) compare rates of learning disabilities in Reading First schools with a comparison group of 
schools; 2) investigate changes in these rates in RF schools before and after grants were 
awarded; and 3) examine the relationship between reading achievement and rates of learning 
disabilities. 

In September 2007, PPSS initiated a follow-up evaluation, the Reading First Implementation 
Evaluation: 2008-09; this study will: 1) describe states' planned responses to reductions of 
federal funding for the program; 2) describe ongoing implementation of the RF program in a 
nationally representative sample of RF districts, and comparable information from a nationally 
representative sample of non-RF Title I districts; 3) conduct case studies of a small and 
purposively selected sample of schools to understand particular aspects of program 
implementation under varying contexts and conditions; and 4) describe individual students' 
reading achievement in RF and non-RF schools as students complete first through sixth grade. 

The U.S. Department of Education’s Institute of Education Sciences is also conducting two 
studies related to Reading First. The Reading First Impact Study is a five-year evaluation 
designed to measure the impact of Reading First on classroom reading instruction and students’ 
reading achievement. That study is conducted in over 240 elementary schools at 1 8 sites and in 
13 states, and is collecting information on students and classrooms in grades 1-3 over a three- 
year period. The Study of Teacher Preparation in Early Reading Instruction is surveying 
preservice teachers at 100 schools of education to answer the question: To what extent does the 
content of teacher preparation programs focus on the essential components of early reading 
instruction? In addition, the study is directly assessing a random sample of 3,000 graduating 
preservice elementary teachers to answer the question: To what extent are graduating preservice 
teachers knowledgeable about the essential components of early reading instruction? 
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The Study Design 



Data Collection 

Two data sources were used to address the study’s evaluation questions: 1) surveys of teachers, 
principals and reading coaches in two nationally representative samples of Reading First schools 
and non-RF Title I schools; and 2) school-level reading scores on state assessments. Exhibit 1.1 
summarizes the primary data collection activities, schedules and samples. The study sample 
included two nationally representative samples of Reading First schools — 550 newly funded 
schools that were in their first year of implementation in 2004-05, and 550 mature schools that 
had been implementing RF activities for one or more years in 2004-05 — along with 550 Title I 
(non-Reading First) schools.*^ The principal, reading coach (if applicable), and one teacher 
(randomly selected) from each of the four target grades of Reading First (K-3) were sent surveys 
to complete. 



Exhibit 1.1 



Data Collection Methods, Samples, and Schedule 




Type of School 


Number of 
Schools 


Estimated Number of 
Respondents 


Schedule 
2004-05 2006-07 


Newly funded Reading First schools 


550 


550 principals 

up to 550 reading coaches 

2,200 teachers 




Mature Reading First schools 


550 


550 principals 

up to 550 reading coaches 

2,200 teachers 




Non-RF Title I schools 


550 


550 principals 

up to 550 reading coaches 

2,200 teachers 





Exhibit reads: 550 newly funded RE sehools were seleeted for inelusion in the study sample. The expeeted respondents 
ineluded 2,200 teaehers, 550 prineipals, and up to 550 reading eoaehes. There were two waves of data eolleetion, one in 
spring 2005 and one in spring 2007. 



Note: Four teaehers per sehool were sampled by randomly seleeting one teaeher from eaeh of grades K-3. One prineipal 
and one reading eoaeh (if applieable) per sehool were surveyed. 



There were two survey data eolleetions, one in eaeh sehool year 2004-05 and 2006-07. Data 
from the 2004-05 data eolleetion were used to address evaluation questions 1 and 2; those data 
deseribe the reading programs in mature RF sehools and eompare them to reading programs in 
non-RF Title I sehools. Findings from analyses of these data were presented in the interim 
report. As displayed in Exhibit 1.1, the 2006-07 data eolleetion ineluded only the sample of 550 
newly funded RF sehools and the sample of Title I sehools. Information from these respondents 



For the purpose of recruiting these two types of RF schools, schools were designated as new or mature based on 
the Reading First program guidelines to states for their annual performance reports; . .for schools receiving 
grants between July 1 and December 3 1 of any reporting period, the current school year will be considered in 
the first year of implementation.” Therefore, schools awarded subgrants before Dec. 31, 2003, were designated 
as mature, because data collection occurred in spring 2005, when those schools were in at least their second 
year of implementation. All other RF schools were classified as newly funded as of the beginning of the first 
data collection in 2005. 
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is used to address question 3 about how reading programs in RF and non-RF Title I schools 
mature over time. 

In early 2005, we recruited 1,649 study schools (1,098 RF schools and 551 non-RF Title I 
schools).^** In the 2004-05 data collection, the response rate across all types of respondents and 
all schools was 96 percent. In the 2006-07 data collection, the corresponding response rate was 
91 percent. (For additional details about response rates see Appendix A Exhibit A.2.) 

The School Sample 

The sample of schools included in the 2007 data collection is a subgroup of schools from the 
original sample of RF and non-RF Title I schools that participated in the first round of data 
collection in 2005. As shown in Exhibit 1.2, RF schools designated as mature — those that had 
been implementing for one or more years in 2005, were not surveyed in 2007. The only RF 
schools surveyed in 2007 were schools that were beginning their implementation of RF activities 
in 2005 (i.e., referred to as ’’newly funded” in the Interim Report). The 2005 sample included a 
total of 1,092 RF schools, 450 mature schools that had been implementing RF activities for one 
or more years, and 642 newly funded schools that were just beginning implementation during the 
2004—05 school year. In 2007, the RF school sample includes 579 schools; these schools have 
now been implementing RF for approximately two years. The reduction of the sample reflects 
the fact some schools closed, were no longer were participating in RF, or did not return any 2007 
surveys. The 2007 non RF Title I sample is 439 schools, reduced from 541 schools (in 2005) for 



Exhibit 1.2 



Respondent Samples of RF and Non-RF Title I Schools in 2005 and 2007 



RF Mature 


RF Newly 


Non-RF 


Total 




Funded 


Title 1 




450 


642 


541 


1,633 



2005 Respondent School 
Sample 



2007 Respondent Status 



Ineligible 
Refused 
Non-respondent 
2007 Respondent Sample 

Weighted 2007 Respondent 
Sample 





(53) 


(65) 


(718) 


Not 

surveyed 


(0) 


(6) 


(6) 


in 2007 


(10) 


(31) 


(41) 




579 


439 


1,018 




1,555 


12,909 


14,464 



Exhibit Reads: The respondent sample in 2005 included 1,092 RF schools (450 mature and 642 newly funded) and 541 
non-RF Title I schools yielding a total school sample of 1,633 schools. 

Notes: The description of the sample in this chapter is based on the 2007 data collection. 

Source: Abt Associates’ Receipt Tracking Files (2005 and 2007). 



Although private schools students and teachers are eligible to receive equitable services under the Reading First 
Program, this evaluation includes only public schools and does not address RF programs implemented in private 
schools. 
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the same set of reasons (See Appendix A, Exhibit A.l, for a summary of the respondent samples 
based on the 2005 recruitment and data collection). 

Analytic Approach 

Analysis of Survey Data 

This report presents findings based on two survey data collections that describe and compare the 
reading programs in RF and non-RF Title I schools. The discussion focuses primarily on 
differences between RF and non-RF Title I schools in 2006-07, although results are displayed 
for both the 2004-05 and 2006-07 data collections. When possible, the same teachers were 
surveyed at each time point. It is important to note that there was substantial turnover 
(approximately 50 percent) in the teacher sample within the study schools between 2005 and 
2007, with teachers either leaving the school or teaching at a different grade level. We selected 
replacement teacher respondents for those teachers who were neither in the same school nor in 
the same grade as they had been in the 2004-05 data collection. The changes in the teacher 
sample mean that teachers’ reports of instruction and program characteristics reflect both the 
evolution of the reading programs over time as well as teacher turnover. There was also 
substantial turnover (approximately 30 percent) in reading coaches from 2004-05 to 2006-07. 

Multiple Hypothesis Testing 

This report presents numerous survey-based findings based on hypothesis tests that compare 
elements of reading programs in RF and non-RF Title I schools in 2005 and 2007. Multiple 
hypothesis testing is of concern because, as the number of tests of statistical significance 
increases so too does the likelihood of falsely claiming there is a difference between two groups 
when there is none (e.g., referred to as a Type 1 error). Our strategy for minimizing such errors 
entails limiting our discussions to only those findings that are large enough to be considered 
meaningful and policy-relevant. The threshold for identifying such meaningful differences 
varies depending upon the specific outcomes being measured. For example, many of the 
outcomes presented in this report are measured in terms percentages. Most often, the report 
highlights differences of 8-10 percentage points or more because such differences are more 
likely to be practically meaningful. Smaller percentage point differences, in fact, are statistically 

significant at the .05 level; however, they are not substantively meaningful and therefore are not 

2 1 

discussed in the text of this report. 

The analyses of the survey responses, in fact, produced meaningful differences that, for the most 
part, were statistically significant at a more stringent threshold than the standard .05 level. In 
general, the meaningful differences presented in this report have a p-value of < .001. By limiting 
the discussion to meaningful and policy-relevant differences between RF and Title I schools, we 
have, in effect simultaneously reduced the chances of committing a Type I error to 1 in 1,000. 
Furthermore, it is often the case that findings based on the 2006-07 data collection are consistent 
(at the p <.001 level) with results reported based on the 2004-05 data collection. For these 



However, all exhibits in this report display statistical significance relying on the standard p-value of p < .05). 
Further, all findings described as ‘significant’ in the text are statistically significant (p-value < .05). 
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findings, we can have greater confidence that they are not due to chance but represent true 
differences between RF and non-RF Title I schools. 

Analysis of Student Reading Achievement 

This report also presents findings based on two analyses of reading achievement trends in RF 
and non-RF Title I schools. One analysis examines third- and fourth-grade reading performance 
on states’ reading assessments. The second analysis examines the relationship between teachers’ 
RF-aligned instructional activities and student reading achievement, using 2005 survey data and 
states’ reading assessment scores. The methods used to conduct these analyses are briefly 
summarized below. 

Reading Achievement on State Assessments 

Analyses were conducted using annual third- (24 states) and fourth-grade (17 states) test scores 
on state reading assessments from 12,362 schools (3,000 RF and 9,362 non-RF Title I). 
Difference-in-difference models were fit to the data to estimate whether the changes in reading 
performance (from pre- to post-RF implementation) were larger in RF schools than in non-RF 
Title I schools. These analyses required that we specify, for each school, those years (and their 
corresponding test scores) that preceded then followed the implementation of RF. Because no 
data provide this information directly, we used four different methods to assign schools’ pre- and 
post-RF implementation years: 1) initial state award date; 2) district award date; 3) an adjusted 
district award date that accounts for the time between district award dates and schools’ receipt of 
funds date; and 4) school award dates based on RF principals’ survey responses. For each of 
the four analyses, we also assigned pre- and post-RF implementation years for the non-RF Title I 
schools (See Appendix C for a discussion of the procedures used to assign award dates). 

Three or four separate methods were fit to the data for each state, depending on data 
availability.^"^ In most states, we fit baseline mean models that estimated the average reading 
performance in the pre-RF years in both RF and non-RF Title I schools, compared that to post- 
RF average performance, and then compared the non-RF Title I pre-post difference to the RF 
pre-post difference (i.e., the difference-in-difference estimate). In addition, for each method, we 
computed a weighted average effect size across states of the pre-RF to post-RF implementation 
differences in reading scores in RF and non-RF Title I schools by averaging weighted effect 
sizes calculated from the difference-in-difference estimate. 

RF-Aligned Activities and Student Reading Achievement 

We used two data sources to investigate the relationship between student reading performance 
and the implementation of RF activities: 1) third-grade reading scores on state assessments (as 

The adjusted distriet award date added an average number of days to the district award date. 

We used these dates to proportionally assign imputed dates to other RF schools in our study sample. 

To conduct these analyses, we must be able to accurately specify, for each school, those test scores that 
represent performance in years prior to as well as in years after RF implementation. Because data on when 
each RF schools received funding are not available, we fit statistical models using four different methods to 
estimate this date. Analyses were run separately for schools within each state; see Appendix C for a detailed 
discussion. 
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used above); and 2) composite variables constructed from a subset of 2005 survey questions that 
characterize classroom reading instruction, help for struggling readers, professional 
development, and the use of assessments. This analysis included only those schools that had 
teacher survey responses and were located in states with sufficient state test score data, and it 
includes both RF and non-RF Title I schools, because the implementation of activities aligned 
with Reading First can potentially occur in either type of school. 

Because tests and metrics vary across states, each state’s reading scores were standardized by 
designating schools as either high- or low-performing relative to all other RF and Title I SWP 
schools in that state.^^ Schools were then selected from the highest and lowest quartiles; the 
analytic sample included 390 schools from 21 states. 

To explore the relationship between teachers’ implementation of RF-aligned activities and 
schools’ performance on states’ third- grade reading assessments, we fit a logistic regression 
model using the four composites to predict the probability that a school scored in the top quartile 
(of all RF and non-RF Title I SWP schools) on their state’s third-grade reading assessment. The 
model also included indicator variables for each state, in order to account for state level variation 
in the probability of a school’s being in the top quartile on the reading assessment and an 
indicator for Reading First or non-RF Title I status. 

Limitations 

This report presents findings based on analyses of survey data and students’ reading performance 
on state assessments. Both sets of findings should be interpreted with caution as the analyses 
face several limitations. 

Most importantly, although the RF and non-RF Title I schools are similar demographically on 
those characteristics measured in the study, (See Chapter 2), schools were not randomly assigned 
to receive RF funding, and therefore there could have been preexisting differences between the 
two groups of schools on unobserved characteristics. We cannot assume that the two groups of 
schools are equivalent but for the fact that one group received RF funds while the other did not. 
This means that the findings cannot support causal inferences that attribute observed differences 
in student reading achievement between RF and non-RF Title I schools to the Reading First 
program. 

Additionally, the survey findings are based on respondents’ self-reported perceptions and 
judgments about the implementation of their reading programs. Moreover, the Reading First 
legislation allows states to provide professional development in scientifically based reading 
instruction to all K-3 teachers, not only the teachers in schools with RF funding; this may reduce 
the potential to find large differences between the reading programs in RF and non-RF Title I 
schools. 



For each state, we selected schools in the lowest quartile (at or below the 25th percentile) and the highest 
quartile (at or above the 75th percentile) on the state’s third grade reading assessment relative to all RF and 
Title I SWP schools. 
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There are several caveats that apply specifically to the analysis of student reading achievement in 
RF schools. First, it is based on states’ average school-level reading assessment scores, and 
because the number of schools varies widely from state to state, this poses an analytic problem. 
California, for example, has approximately 800 RF schools and nearly 100,000 K-3 students, 
whereas Delaware has eight RF schools and fewer than 1,000 K-3 students. The ability to detect 
statistically significant differences decreases as the number of schools within a state decreases. 
Second, year-to-year comparisons of performance are based on different cohorts of children, who 
may or may not be comparable over time. Therefore, changes in performance could reflect 
individual student differences, rather than exposure to Reading First. 

Third, most states report reading scores in terms of the percent of students meeting a particular 
cutoff or reference point (i.e., percent at or above basic level), not in terms of each school’s 
average test score. Such proficiency scores mask any student level changes in performance that 
may have occurred above or below the cutoff point. This is particularly worrisome in low- 
performing schools and for low-performing students; even substantial changes in student 
achievement can go undetected when average student performance does not cross a given 
threshold. Data on the school-level percent meeting a threshold measure may, therefore, 
underestimate improvements in student performance. 

Fourth, we did not have accurate data on when each school began to implement RF. Such data 
are essential for determining schools’ pre- and post-RF implementation years. Absent such 
information, we used several methods to assign start dates and thereby classify pre- and post-RF 
implementation years. The extent to which such dates were imputed varied depending on the 
analytic method. For example, the analysis that used the initial state award date required no 
imputation because this method defines one start date for the onset of RF and this date is applied 
to all RF and non-RF Title I schools in the analysis. Also, for the analyses that used the dates RF 
districts received their funds, there was no imputation for the RF and non-RF Title I schools in 
those districts because, again, those dates define the onset of RF implementation. However, 60 
percent of the non-RF Title I schools are in districts with no RF schools and therefore, no RF 
district start dates. For those schools, we used dates from RF districts to impute schools’ start 
dates. However, these imputations may have led us to misclassify pre- and post-RF years for 
some schools, and consequently may have affected the results of these analyses. 

There are also two limitations specific to the investigation of the relationship between reading 
achievement and teachers’ implementation of RF-aligned activities. First, the sample of schools 
included in the analysis is a sample of convenience, despite the fact that it is drawn from 
nationally representative samples of RF and non-RF Title I schools, because it includes only 
those schools with survey data and data from their states’ third-grade reading assessments. 
Therefore, we must caution against generalizing any observed findings to larger populations of 
schools. 

Second, the relational analysis uses measures that are differentially sensitive; the teacher survey 
asks detailed questions about RF activities, and the achievement measure is a blunt assessment of 
third-grade reading performance. Further, the school sample includes only the extremes (highest 
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and lowest quartiles) of the reading score distributions. We constructed a dichotomous (1/0) 
variable to indicate whether a school’s score was in the top quartile to try to standardize reading 
achievement across states as best we could, given the available data. However, this measure of 
reading achievement may not capture all schools’ improved reading achievement. 

Summary of Findings from Reievant Reports 

To place the current findings in context, below we review the key findings from several other 
relevant evaluations. First, we describe earlier (interim) findings from this study. Second, we 
describe other studies of the Reading First Program, and, third, we describe two studies about 
reading programs and instruction that have clear implications for Reading First. 

The Reading First Impiementation Evaiuation: Interim Report (U.S. Department of Education, 

2006) 

The Interim Report^^ presented findings that described and compared reading programs 
implemented in 2004-05 in nationally representative samples of RF and non-RF Title I schools. 
These findings provide the baseline for understanding how these programs have changed by 
2006-07, the focus of this current report. 

The main finding from this report is that Reading First schools appear to be implementing the 
major elements of the program as intended by the legislation, such as providing scientifically 
based reading instruction in grades K-3, increased amounts of time for reading instruction, 
interventions for struggling readers, wider use of classroom-based reading assessments, and 
more professional development activities. 

Reading Instruction in K-3 Classrooms 

• Instructional Time. Significantly more RF schools than non-RF Title I schools reported 
having a formally scheduled “reading block for teaching reading for each of grades K 
through three. Teachers in RF schools reported, on average, they spent significantly 
more time on reading than did teachers in non-RF Title I schools — a difference of about 
19 minutes per day, or almost 100 minutes per week. Teachers in newly funded RF 
schools were also significantly more likely than teachers in non-RF Title I schools to 
report that they had increased the amount of time spent on reading from the 2003-04 to 
the 2004-05 school years. 

• Instructional Materials. Newly funded RF schools reported that they have made 
substantial changes to their reading programs since they received their RF funds in the 
2004—05 school year. 

• Instructional Strategies. Teachers in RF schools across all grades rated a greater 
proportion of SBRR-aligned practices as central to their instruction than did teachers in 
non-RF Title I schools. 



This report is available at http://www.ed.gov/about/offices/list/opepd/ppss/html. 
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Interventions for Struggling Readers 



• RF teachers in three grades (kindergarten, second, and third) were significantly more 
likely than teachers in non-RF Title I schools to place their struggling students in 
intervention programs. 

• RF teachers were more likely than teachers in non-RF Title I schools to report increased 
amounts of time for struggling readers to practice skills in several dimensions of reading 

• Across both RF and non-RF Title I schools, there is evidence of increased attention to 
struggling readers, including, for example, no delay between identification and provision 
of services. RF and non-RF Title I schools are also similar with respect to coordinating 
instruction for ELL students 

• Although significantly more teachers in RF schools reported receiving professional 
development in helping struggling readers than did teachers in non-RF Title I schools, 
teachers in both RF and non-RF Title I schools recognized the challenge of providing 
effective instruction to struggling readers; 80 percent of teachers in both groups reported 
that they need additional professional development on this topic. 

Assessment 

• Teachers in RF schools were more likely than teachers in non-RF Title I schools to report 
having received professional development assistance on administering and using 
assessments, interpreting assessment data, and using diagnostic tests to guide instruction. 

• Teachers in RF schools were more likely to report applying assessment results for varied 
instructional purposes (e.g., for planning the grouping of students, progress monitoring 
and identifying struggling readers) than their non-RF Title I counterparts. 

Oversight and Classroom Support Activities 

• RF schools were significantly more likely to have a reading coach than were non-RF 
Title I schools. 

• Coaches in RF schools were significantly more likely to provide teachers with various 
supports for their reading instruction than were coaches in non-RF Title I schools. 

Professional Development 

• RF staff received significantly more professional development related to reading than did 
staff in non-RF Title I schools. Teachers in RF schools were more likely to have 
received professional development in the five dimensions of reading instruction as well 
as in overall teaching strategies than teachers in non-RF Title I schools. Indeed, teachers 
in RF schools reported feeling better prepared to teach the five dimensions of reading 
than teachers in non-RF Title I schools. 
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• Activities attended by teachers in RF schools were more likely to have eharacteristics 
conducive to a suecessful experienee, such as incentives and follow-up activities, than 
those attended by teachers in non-RF Title I schools. 

The Reading First Impact Study: interim Report (U.S. Department of Education, 2008) 

97 

The Reading First Impaet Study is designed to answer questions about the impact of the 
Reading First Program on classroom instruction and student reading achievement as well as 
about the relationship between instruction and reading achievement. The study colleeted data in 
248 sehools loeated at 18 sites and in 13 states over three school years. This data collection 
included detailed elassroom observations in grades 1 and 2 and reading eomprehension 
assessments in grades 1, 2 and 3. The study uses a regression diseontinuity design to compare 
two groups of schools: those that did and those that did not receive Reading First funding. The 
study’s recently released Interim Report presented findings based on the first two of three years 
of data colleetion. 

• On average, Reading First increased instructional time spent in the reading bloek on the 
five essential components of reading instruetion promoted by the program (phonemic 
awareness, phonies, vocabulary, fluency, and comprehension). 

• On average, aeross the 18 partieipating sites (17 school districts and one statewide 
program), estimated impacts on student reading eomprehension test scores were not 
statistieally significant. 

• Average impacts on reading comprehension and classroom instruction did not change 
systematically over time as sites gained experience with Reading First. 

The Analysis of State K-3 Reading Standards and Assessments (U.S. Department of Education, 
2005) 

98 

This report presented findings from an analysis of the alignment of state K-3 reading eontent 
standards in 2002-03 with the five essential components of reading instruetion in a random 
sample of 20 states. In addition, the study reviewed Reading First applieations of all states and 
the District of Columbia to determine the extent to which state assessments for NCLB 
accountability purposes were used to measure Reading First outcomes. 

• Comprehension and, to a lesser extent, voeabulary were more eomprehensively addressed 
by states’ K-3 reading standards than were the other three essential components of 
reading instruetion (phonemic awareness, phonies and fluency). 

o Reading eomprehension was the most represented of the five essential eomponents in 
the state K-3 reading standards with an average of 57 standards per state, followed by 
vocabulary (19), phonics (16), fluency (6), and phonemic awareness (6). 
o Most standards representing each of the essential eomponents were judged to be at 
the appropriate grade for most states. 

The report available at http://ncee.ed.gov. 

The report is available at http://www.ed.gov/about/offices/list/opepd/ppss/html. 
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• With the possible exeeption of vocabulary and comprehension in grade 3, statewide 
reading assessments in 2003-04 did not significantly address expected student outcomes 
from reading instruction in the five essential components. 

o Twenty states identified their grade 3 reading assessments in 2003-04 as measures of 
Reading First outcomes, primarily for only vocabulary and comprehension. 

Reading First, Locaiiy Appreciated, Nationaiiy Troubied (Center on Education Poiicy, 2007) 

This report^*’ was based on survey data from 50 states and 349 Reading First school districts, and 
case study interviews in nine Reading First districts. It provided information about the perceived 
effectiveness of Reading First, implementation challenges, coordination between the RF and 
Title I programs, and the efficacy of the evaluations of Reading First. Key findings include: 

• Schools and districts appeared to be implementing Reading First as intended by the 
legislation. 

• Two-thirds of school district respondents reported that reading instruction had changed in 
RF schools. 

• Most states and the majority of school districts reported that Reading First improved 
student reading achievement. District respondents attributed improved reading 
achievement to the assessments and instructional programs required by Reading First. 

• More than half of the RF districts reported that schools other than those receiving RF 
funds were modifying their reading programs to align with the key features of Reading 
First. 

Closing the Reading Gap: Findings from a Randomized Trial of Four Reading Interventions for 
Striving Readers (Torgeson, et al., 2006) 

This report^” described the implementation and impacts of four remedial reading interventions on 
the skills of third- and fifth-grade struggling readers. The interventions are Corrective Reading, 
Failure Free Reading, Spell Read Phonological Auditory Training and Wilson Reading. These 
interventions provided explicit and systematic instruction in the component skills of reading. 

The study design included randomly assigning teachers to one of the four programs, as well as 
randomly assigning students to one of the programs or to a control group where students 
received the normal instruction provided by the school. Comparing the combined effects of the 
four interventions to the control group, the study found that: 

• There were positive impacts on third-grade students’ skills on decoding, word reading 
accuracy, fluency and passage comprehension, and on fifth-grade students’ decoding 
skills: 



29 

30 
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The report is available at http://www.cep-dc.org. 

The report is available at http://ies.ed.gov/ncee/pubs/20084012/index.asp. 
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• Both third- and fifth-grade students receiving the interventions improved their 
performance on a measure of decoding skills (i.e., word attack) relative to the 
performance of the normative sample of students for that particular test. 

Instructional Time in Elementary Schools: A Closer Look at Changes for Specific Subjects 
(Center on Education Policy, 2008) 

This study^* analyzed data from the surveys of 349 school districts (described in the “Reading 
First: Locally Appreciated, Nationally Troubled” report) to examine changes in instructional 
time in elementary schools from the time NCLB took effect in 2002 to 2007. 

• About 80 percent of districts reported increasing the amount of time devoted to English 
and language arts in their elementary schools. 

• According to district reports, on average, elementary schools have increased by about 
three hours per week the amount of time spent teaching English, language arts and 
mathematics. 

The Organization of This Report 

The report is organized into nine chapters. Chapter 1 reviewed the context for study, the 
Reading First legislation, the study design and the analytic approaches. Chapter 1 also 
summarized findings from other relevant studies. Chapter 2 describes the characteristics of RF 
and non-RF Title I schools. Chapters 3 through 7 present analyses of survey data that investigate 
changes in reading programs from 2005 to 2007 in RF and non-RF Title I schools. Chapter 8 
presents findings about reading achievement trends in RF and non-RF Title I schools based on 
state-by-state analyses of third- and fourth-grade scores on states’ reading assessments. In 
addition, chapter 8 presents results of an analysis of the relationship between teachers’ RF- 
aligned activities and student reading achievement using 2005 survey data and states’ reading 
assessment scores. Chapter 9 presents conclusions based on the key findings presented in this 
report. 



The report is available at http://www.cep-dc.org. 
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Chapter 2: Characteristics of RF and Non-RF Titie I 
Schoois 



This chapter presents information on school enrollment, staffing, student populations, and 
external resources targeted to schools’ reading programs. Finally, we summarize the proportion 
of RF schools designated as in need of improvement as defined by the NCLB accountability 
requirements. 

School Characteristics 

RF and non-RF Title I schools are similar in terms of their average enrollment and location 
(urbanicity) (Exhibit 2.1). However, there is a modestly greater proportion of very large RF 
schools than non-RF Title I schools (14 percent vs. 9 percent). There are no significant or 
substantive differences in the locales of RF and non-RF Title I schools. 



Exhibit 2.1 



School Enrollment and Urbanicity in RF and Non-RF Title I Schools, 2006-07 School 
Year 





RF Schools 


Non-RF Title 1 Schools 


School Size 


Mean enrollment 


484 


454 




Percent 


Percent 


Very small (1-99) 


4% 


3% 


Small (100-249) 


13 


16 


Medium (250-499) 


41 


44 


Large (500-749) 


29 


28 


Very large (750+) 


14* 


9 


Urbanicity 


Urban 


40% 


36% 


Suburban 


35 


35 


Rural 


25 


29 



Exhibit reads: In 2006-07, about 4 percent of RF and 3 percent of non-RF Title I schools had very small 
enrollments (i.e., less than 100 students). 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and 
non-RF Title I schools. 

See Appendix B, Exhibit B.2.2 for additional statistics. 

Source: Principal Survey, Question A3. 

Weight: Principal. 

Weighted respondents: Principals in 1,536 RF schools and 12,802 in non-RF Title I schools. 

Nonresponse rates across survey items: 0.8 to 6.0 percent. 



Enrollment stability is similar for RF and non-RF Title I schools, with approximately 30 percent 
of the schools in each group experiencing an increase in enrollment; about 40 percent of the 
schools experienced a decrease in enrollment over the last five years (Exhibit 2.2). Reading First 
and non-RF Title I schools also have similar attendance rates and mobility rates (93 percent and 
17-18 percent, respectively). 
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Exhibit 2.2 



Mobility Rates, Attendance Rates and Changes in Enrollment in RF and Non-RF Title I 
Schools, 2006-07 School Year 





RF Schoois 


Non-RF Titie i Schoois 


Change in Enrollment in Last Five Years 

Decreased 


40% 


39% 


Remained stable 


31 


28 


Increased 


29 


33 


Mobility Rate 


18% 


17% 



Attendance Rate 93 93 

Exhibit reads: In 2006-07, attendance rates in both RF and non-RF Title I schools were, on average, 93 percent. 
Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non- 
RF Title I schools. 

See Appendix B, Exhibit B.2.2 for additional statistics. 

Source: Principal Survey, Questions A3 and A4. 

Weight: Principal. 

Weighted respondents: Principals in 1,540 RF schools and 12,719 non-RF Title I schools. 

Nonresponse rates across survey items: 0 to 13.3 percent. 



Staff Experience 

Staff reports of their years of experience are, in general, similar in RF and non-RF Title I schools 
(Exhibit 2.3). Principals in non-RF Title 1 schools have slightly more experience as principals 
than do their counterparts in RF schools (8.2 years vs. 7.2 years). Similarly, they have been in 
their current schools slightly longer than principals in RF schools (5.7 years vs. 4.8 years). 

About half of the principals, on average, have been in their schools for three years or less (5 1 
percent of the RF principals and 44 percent of the non-RF Title 1 principals). 

While teachers in both RF and non-RF Title I schools are very experienced, teachers in non-RF 
Title I schools, on average, have slightly more experience than do teachers in RF schools (15.5 
years vs. 14.1 years). Nevertheless, about 19 percent of RF teachers and 16 percent of teachers in 
non-RF Title 1 schools have taught in their schools for three years or less (See Appendix B.2.3b 
for details). 
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Exhibit 2.3 



Years of Experience for Staff in RF and Non-RF Title I Schools, 


2006-07 School Year 




RF Schools 


Non-RF Title 1 Schools 




Mean 


Mean 


Principals 


Years experience as principal 


7.2 


8.2* 


Years in this school 


4.8 


5.7* 


Teachers 


Years experience 


14.1 


15.5* 


Years in this school 


9.3 


10.9* 


Reading Coaches ^ 


Years experience 


17.7 


16.8 


Years in this school 


9.7 


9.3 


Years as reading coach in this school 


3.3 


4.5 




Percent 


Percent 


Schools with reading coaches 


99* 


57 


Principals in this school three years or 


51* 


44 



less 

Exhibit reads: In 2006-07, principals in RE schools have, on average, 7.7 years experience in that position, 
compared with 8.5 years for principals in non-RE Title I schools. This difference is statistically significant (p < 
.05). 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and 
non-RF Title I respondents. 

^ reading coaches in non-RF Title I schools were excluded from this analysis because based on their survey 
responses, they do not appear to meet the definition of “reading coach” used in this evaluation. 

See Appendix B, Exhibit B.2.3a for additional statistics. 

Source: Principal, and Teacher Surveys, Question Al; Reading Coach Survey, Question A3. 

Weight: Principal, Teacher, and Reading coach. 

Weighted respondents: 1,555 principals, 5,811 teachers and 1.533 reading coaches in RF schools; 12,909 
principals, 45,731 teachers and 5,798 reading coaches in non-RF Title I schools; 

Nonresponse rates across survey items and respondents: < 1 percent. 



Nearly all RF schools have reading coaches (99 percent), compared to 57 percent of non-RF 
Title I schools. The reading coaches in both RF and non-RF Title I schools are more 
experienced than teachers in these schools; on average, they have 18 years of combined teaching 
or coaching experience (Exhibit 2.4). 

Preservice Teacher Training in the Five Dimensions of Reading 

A cornerstone of the Reading First program is that teachers should be knowledgeable about and 
well prepared to teach the five essential components of reading instruction — ^phonemic 
awareness, phonics, vocabulary, comprehension, and reading fluency. Teachers rated the extent 
to which their preservice training prepared them to teach the five dimensions of reading using a 
scale of 1 (not at all prepared) to 5 (extremely well-prepared). Generally, across the five 
dimensions, RF teachers rated themselves in the middle of the range — suggesting that their 
preservice training left them somewhat prepared to teach these skills. On two dimensions 
(vocabulary and fluency), teachers in non-RF Title I schools rated themselves significantly 
higher than did the RF teachers; these differences, while statistically significant, are small 
(Exhibit 2.4). 



Chapter 2: Characteristics of RF and Non-RF Titie i Schoois 



19 




Exhibit 2.4 

Preservice Training on the Five Dimensions of Reading: Teachers’ Ratings on 
Preparedness, 2006-07 School Year 



Teachers 



Dimension 


RF Schoois 


Non-RF Titie i Schoois 


Mean Rating 


Mean Rating 


Phonemic awareness 


3.07 


3.11 


Phonics 


3.19 


3.22 


Vocabulary 


3.34 


3.45* 


Comprehension 


3.41 


3.50 


Fluency 


2.95 


3.07* 



Exhibit reads: In 2006-07, teachers in non-RE Title I schools rated themselves as better prepared based on 
their preservice training to teach vocabulary than did RE teachers (3.45 vs. 3.34, p < .05). 

Notes: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between 
teachers in RF and non-RE Title I schools. 

Ratings are on a scale of 1 to 5, where 1 = not at all prepared and 5 = extremely well prepared. 

See Appendix B, Exhibit B.2.4 for additional statistics. 

Source: Teacher Survey, Question A3. 

Weight: Teacher. 

Weighted respondents: 23,880 teachers in RE schools; 180,630 teachers in non-RE Title I schools. 
Nonresponse rates across survey items: < 1 percent. 



Student Population 

Principals reported that special education services are provided to the same proportion of 
students in RF and non-RF Title I schools (8 percent) (Exhibit 2.5). A modestly larger 
proportion of students reeeive English as a Seeond Language (ESL) edueation services in RF 
schools than in non-RF Title I sehools (18 pereent vs. 1 1 pereent). Very small proportions of 
students in RF and non-RF Title I schools receive instruction in a language other than English (4 
pereent and 6 percent, respectively). Also, slightly more students in non-RF Title I schools are 
reportedly reading at or above grade level than their eounterparts in RF sehools (60 pereent vs. 
54 percent). 
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Exhibit 2.5 

Student Characteristics in RF and Non-RF Title I Schools, 


2006-07 School Year 




RF Schools 


Non-RF Title I 
Schools 




Mean Percent 


Mean Percent 


Receive Special Education Services 


8 


8 


Receive ESL Instruction 


18* 


11 


Instruction in language other than English 


6 


4 


Reading at or above grade level 


54 


60* 



Exhibit reads: In 2006-07, in RF schools and non-RF Title I schools, 6 percent of kindergarten students 
receive special education services. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF 
and non-RF Title I schools. 

See Appendix B, Exhibit B.2.5 for additional statistics. 

Source: Principal Survey, Question A5. 

Weight: Principal. 

Weighted Respondents: Principals in 1,446 RF schools and 1 1,460 in non-RF Title schools. 

Nonresponse rates across survey items and grades: 4.5 to 15.0 percent. 



External Resources to Support Schools’ Reading Programs 

Reading First is the signature reading program of No Child Left Behind. As such, it represents a 
substantial investment aimed at improving the reading achievement of the nation’s students. At 
the same time however, states, school districts, and schools receive support for their reading 
programs from other sources. As noted in the Interim Report, interviews with state Reading First 
coordinators, for example, indicated that that 30 states had other reading initiatives based on 
scientifically based reading research (SBRR), 36 states had major statewide initiatives that 
provided reading-related professional development to teachers and other educators. Below, we 
summarize survey results about the array of funding and external support for reading for Reading 
First schools. 

Size of Reading First Grant 

The median grant amount for RF schools in 2004-05 was $138,000 (Exhibit 2.6). Based on 
principal survey responses, newly funded RF schools in their first year of implementation 
received significantly more RF funds, on average, than more mature schools that had been 
implementing for one or more years ($168,000 vs. $120,690). These differences are also 
reflected in the per pupil allocations of RF funds; the median per pupil allocation of RF funds 
was significantly higher in newly funded schools than in mature schools in 2004-05 ($670 vs. 
$403). 



Other initiatives include revised reading or language arts standards and accountability or assessment initiatives 
focused on reading proficiency (29 states), early child education and school readiness initiatives (18 states), and 
family literacy programs such as Even Start. 
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Exhibit 2.6 

Size of Reading First Grant, Per School and Per Pupil, 2004-05 School Year 




Size of RF grant 


All Reading First 
Schools 


Newly Funded 
Reading First 
Schools 


Mature Reading 
First Schools 


Median Reading First school grant 


$138,000 


$168,000* 


$120,690 


Median per pupil grant 


$496 


$670* 


$403 



Exhibit reads: In 2004-05, the median amount of sehools’ RE grants was $138,000; for newly funded RE sehools, the median 
grant was $168,000 and for mature RE sehools the grant was $120,000. 

Note: The stars (*) indieate that there is a statistieally signifieant differenee (i.e., p-value < .05) between RF and non-RE Title I 
sehools. 

See Appendix B, Exhibit B.2.6 for additional statisties. 

Souree: Prineipal Survey (2005), Questions A4 and B6. 

Weight: Prineipal. 

Weighted Respondents: Prineipals in 2,539 RF sehools. 

Nonresponse rate: 35.1 pereent. 



Sources of Financial Assistance for K-3 Reading Programs 

Principals of RF schools, on average, reported that they received funding from more sources than 
did principals of non-RF Title I schools (5.2 vs. 4.5) (Appendix B, Exhibit B.2.7a). However, a 
significantly greater proportion of principals in non-RF Title I schools than RF schools reported 
receiving funding from non-RF Title I, district general funds, state textbook funds, and private 
grants than did principals in RF schools. 

According to principals, virtually all RF schools (97 percent) received Title I funds in 2006-07. 
One might ask, therefore, whether schools receiving RF funds receive smaller allocations of 
Title I funds than similar compared to other non-RF Title I schools. Findings from the National 
Longitudinal Study ofNCLB (U.S. Department of Education, 2008) suggest otherwise. 

Analyzing Title I funding allocations in 39 districts and 458 RF schools indicated that nearly all 
districts either gave similar or larger Title I allocations per low-income student to their RF 
schools compared with non-RF Title I schools. Interestingly, schools’ Reading First allocations 
were, on average, substantially smaller than their Title I allocations ($269 per low-income 
student for Reading First vs. $731 for Title I schools). However, there were 79 schools (18 
percent) whose Reading First allocations per low-income student were larger than their Title I 
allocations. 

Sources of Nonfinancial Assistance for K-3 Reading Programs 

Beyond financial support, RF principals were significantly more likely to report receiving 
substantially more external, nonfinancial assistance than were principals in non-RF Title I 
schools in a variety of areas; examples include conducting classroom observations (73 percent 
vs. 46 percent), reviewing reading program effectiveness (64 percent vs. 42 percent), providing 
technical assistance for using supplementary reading materials (61 percent vs. 47 percent and 
planning professional development (76 percent vs. 65 percent) (Exhibit 2.7). 
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Exhibit 2.7 






Nonfinancial External Assistance for K-3 Reading Program Activities in RF and Non-RF Title 1 
Schools, 2006-07 School Year 


Type of Assistance 


RF Schools 


Non-RF Title I 
Schools 


Planning professional development 


76%* 


65% 


Interpreting assessment results 


75 * 


65 


Conducting classroom observations 


73* 


46 


Providing technical assistance in implementing core reading 
programs 


65* 


45 


Selecting professional development providers 


63* 


49 


Selecting assessment instruments 


54 


54 


Selecting instructional programs/materials 


52 


53 


Reviewing reading program effectiveness 


64* 


42 


Conducting demonstration lessons 


57* 


45 


Diagnosing needs of struggling readers 


58* 


51 


Setting up intervention programs for struggling readers 


58 


52 


Providing technical assistance for using supplementary reading 
materials 


61* 


47 


Conducting needs assessment for professional development 


48* 


36 


Leading teacher study groups 


37* 


29 


Recruiting staff with reading expertise 


28 


24 



Exhibit reads: In 2006-07 , 16 percent of principals in RF schools reported receiving external assistance in planning professional 
development, compared to 65 percent of principals in non-RE Title I schools. This difference is statistically significant (p-value < 
.05). 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.2.7 for additional statistics. 

Source: Principal Survey, Question B2. 

Weight: Principal. 

Weighted respondents: Principals in 1,523 RF schools and 12,635 non-RF Title schools. 

Nonresponse rates across survey items: 0.7 to 1.2 percent. 



NCLB Accountability 

All schools, including Reading First schools, must also continue to meet the aecountability 
requirements of the NCLB legislation. NCLB mandates that states develop and implement 
accountability systems to ensure that distriets and schools make adequate yearly progress as 
measured by the aeademie achievement of their students (Part A, See. 1111, (b), (2)). States 
develop definitions of Adequate Yearly Progress (AYP) for their respeetive distriets and sehools 
by speeifying minimum levels of improvement in student performanee to be attained. In 2006- 
07, prineipals in 72 pereent of RF schools and 78 percent of non-RF Title I schools reported that 
their schools made AYP in reading or language arts based on 2005-06 test scores. 

Sehools that do not make AYP for two consecutive years are designated as schools in need of 
improvement; such schools are required to develop plans for improvement and offer sehool 
choiee in year 1 of school improvement. If, however, sehools fail to make AYP at the end of 
year 1 or improvement status, distriets are required to provide technieal assistance to the school 
and supplemental educational services to the eligible students in those sehools. If those sehools 
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fail to make AYP at the end of year 2 of improvement, the district must implement a series of 
corrective actions such as replacing staff, restructuring the organization of the school, 
implementing new curricula, or extending the school day. 

Schools are in restructuring status after failing to make AYP after five years. In the first 
restructuring year, the district must develop a plan for schools that do not make AYP in the sixth 
year to implement at the beginning of the following school year. Schools can: reopen as a 
charter school; replace all or most of the school staff; contract with a private entity to manage the 
school; turn over operation of the school to the state; or adopt some other major restructuring of 
the school’s governance {The No Child Left Behind Act of 2001, PL 107-110, Section 1 1 16(b) 
(8)). Findings from a recent study of Title I found that the majority of schools in the restructuring 
phase are undergoing other major restructuring of their school governance practices and are 
implementing interventions that are corrective actions rather than restructuring actions (Stullich, 
Eisner and McCrary, 2007). 

Reading First Schools in Need of Improvement 

In the 2006-07 school year 32 percent of RF schools were designated as schools in need of 
improvement (SINI) (Exhibit 2.8), an increase schools from the 22 percent in 2004-05. This 
proportion is consistent, however, with recent data from the Final Report on the National 
Assessment of Title I Final Report indicating that in 2005-06 about one-third of high-poverty 
Title I schools were designated as in need of improvement (U.S. Department of Education, 
2007 ).^^ 



Exhibit 2.8 



Status of Reading First Schools Designated as in Need of Improvement, 2004-05 and 
2006-07 School Years 





2004-05 


2006- 


-07 




Number 


Percent 


Number 


Percent 


Reading First Schools 


4,764 


100 


5,115 


100 


Schools in need of improvement 


1,014 


22 


1,632 


32 


Status: 










School improvement — Year 1 


418 


41 


444 


27 


School improvement — Year 2 


294 


29 


323 


20 


Corrective action 


189 


19 


481 


29 


Restructuring and planning 


113 


11 


384 


24 


Exhibit Reads: In 2004-05, of the 4.764 RF sehools, 1,014 (22 pereent) were designated as in need of improvement. 
Note: These data are based on the eensus of Reading First sehools rather than the study’s sample of RF sehools. 
Souree: Databases provided by the U.S. Department of Edueation. 



High poverty schools were defined as schools in which at least 75 percent of the students are eligible for free or 
reduced price lunches. These schools, although not an ideal comparison with RF schools, represent the most 
reasonable group for which such data are available. 
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Summary 



The demographic characteristics of RF and non-RF Title I schools are similar in several areas 
including attendance rates, mobility, and stability of enrollment. However, RF schools are more 
likely to be very large (i.e., enrollments of 750 or more students) than are non-RF Title I schools 
(14 percent vs. 9 percent). 

The K-3 student populations of RF and non-RF Title I schools are similar in terms of the 
proportion of students receiving special education services and instruction in a language other 
than English. However, these schools differ in their proportions of students receiving ESL 
instruction; about 18 percent of K-3 students in RF schools receive such services, compared to 
about 1 1 percent in non-RF Title I schools. Also, slightly more students are reading at or above 
grade level in non-RF Title I schools than in RF schools (60 percent vs. 54 percent). 

In addition to receiving Reading First funds, most RF schools received support for reading 
programs from several other sources including Title I (91 percent), school district general funds 
(82 percent), state funds for reading programs (56 percent) and state textbook funds (45 percent). 
Reading First schools also received many different kinds of nonfinancial support for their K-3 
reading programs in the form of assistance with planning professional development, interpreting 
assessment results, implementing the core reading program, and conducting classroom 
observations. Finally, across a variety of types of nonfinancial support, Reading First schools 
were much more likely to receive such assistance than were non-RF Title I schools. 
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Chapter 3: Reading Instruction 

A central objective of Reading First is to improve how reading is taught in K-3 classrooms by 
aligning instruction with scientifically based reading research (SBRR). Ongoing professional 
development, the use of assessments, and interventions for struggling readers are all aimed at 
making reading instruction more effective, thereby increasing the proportion of children who can 
read at grade level. 

This chapter presents findings that describe and compare the specific components of reading 
programs in RF and non-RF Title I schools, including the amount of instructional time allotted to 
reading instruction, the types of materials used for reading instruction, and the specific strategies 
and activities teachers use to teach their students to read. 

Key Findings 

RF schools reported dedicating more time to reading instruction in their K-3 classrooms 
than did non-RF Title I schools. According to principals and reading coaches, virtually all 
schools scheduled reading blocks for their K-3 classrooms. Nearly all (98 percent) of RF 
schools scheduled these blocks for 90 minutes or more, compared to 75 percent of non-RF Title I 
schools. Moreover, K-3 teachers in RF schools reported spending an average of 103 minutes per 
day on reading activities, compared to 81 minutes reported by teachers in non-RF Title I schools. 
This translates to approximately 110 additional minutes per week of reading instruction for 
students in RF schools. 

Reading materials in RF schools were more likely to be aligned with SBRR than in non-RF 
Title I schools. According to reading coaches, RF schools were more likely to report that their 
core reading programs are aligned with SBRR (93 percent vs. 76 percent), that K-3 teachers in 
their schools are knowledgeable about SBRR (79 percent vs. 58 percent) and that reading 
intervention materials in their schools are aligned with SBRR (94 percent vs. 79 percent) than 
were their counterparts in non-RF Title I schools, hr addition, more teachers in RF schools 
teachers in non-RF Title 1 schools rated SBRR-aligned practices as central to their instruction (85 
percent vs. 75 percent). 

Virtually all RF and non-RF Title I schools implemented at least one core reading program 
in each of grades K-3. RF schools were more likely than non-RF Title I schools to report that 
they the used the same program across all of their K-3 classrooms (78 percent vs. 58 percent). 
The five most frequently used core reading programs in RF schools included Harcourt Trophies 
(23 percent), McGraw-Hill Open Court (15 percent), Scott Foresman Reading (13 percent), 
Houghton Mifflin Nation’s Choice (1 1 percent) and Houghton Mifflin’s Reading (10 percent). 
Four of these five programs were also those most commonly reported by non-RF Title I schools 
(Houghton Mifflin’s Nation’s Choice was not). 



Chapter 3: Reading Instruction 



27 





After making substantial changes to their reading programs during their first year of 
implementation, RF schools were less likely than non-RF Title I schools, to make many 
additional changes in 2006-07. In mature RF schools, 3 percent adopted a new core reading 
program in 2006-07 compared to 39 percent in 2004-05; similarly, 14 percent added new 
materials for ELLs, compared to 43 percent earlier. RF schools continued to add new 
intervention programs for struggling readers (40 percent) and add new supplementary materials 
to their reading programs (42 percent) in 2006-07. In 2006-07, non-RF Title I schools were 
more likely than RF schools to report they adopted a new core reading program (17 percent vs. 3 
percent) and adopted new materials for ELLs (26 percent vs. 14 percent). In both years, about 60 
percent of non-RF Title I schools reported that they added new supplementary materials and 
more than 40 percent reported adding new interventions for struggling readers. 

Instructional Time 

Building upon research demonstrating that the amount of time schools spend on reading is a 
major determinant of reading achievement,^"^ the Reading First program guidance encourages 
schools to “consider the allocation of time, including a protected, uninterrupted block of time for 
reading instruction of more than 90 minutes per day.”^^ Findings from surveys of both 
administrators and teachers suggest that schools receiving Reading First funds are adhering to 
the guidance and devoting more classroom time to reading instruction than are non-RF Title I 
schools. 

Reading coaches in both RF and non-RF Title I schools generally agreed that sufficient time 
during the school day was allotted for reading instruction, however, the actual amount of reading 
instruction reported was significantly greater in RF schools than non-RF Title I schools (Exhibit 
3.1). Virtually all RF schools and most non-RF Title I schools scheduled formal reading 
blocks for grades K-3 (99 vs. 91 percent for kindergarten, 100 vs. 94 percent for grades 1-3), but 
the average length of these reading blocks was 12-14 minutes longer in RF schools. In fact, 
virtually all RF schools reported that their scheduled grade 1-3 reading blocks last 90 minutes or 
more, whereas 20 percent of non-RF Title I schools reported having reading blocks shorter than 
90 minutes for these grades. 



Committee on the Prevention of Reading Difficulties in Young Children, National Research Council, 
Washington, D.C., 1998. 

U.S. Department of Education, Office of Elementary and Secondary Education, Guidance for the Reading First 
Program. Washington, D.C., April 2002, page 6 

When asked to indicate on a scale of one to five how accurately the statement “Sufficient time during the school 
day is allotted for reading instruction” describes their school, 94 percent of the reading coaches in RF schools 
and 85 percent of the reading coaches in non-RF Title I schools said this statement was “accurate” or “very 
accurate.” This difference is statistically significant (p < .05). 
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Exhibit 3.1 



Daily Scheduled Reading Block and Mean Length of the Reading Block for RF and Non-RF Title I 
Schools, 2004-05 and 2006-07 School Years 





2004-05 


2006-07 


2006-07 


Grade Level 


RF 

Schools 


Non-RF 
Title 1 
Schools 


RF 

Schools 


Non-RF 
Title 1 
Schools 


Difference 
- 2004-05 
Difference 


Kindergarten 

...Schools with reading block 


99%* 


88% 


99%* 


91% 


-2.6% 


...Schools with a reading block < 90 minutes 


9% 


32%* 


7% 


31 %* 


-0.3% 


...Length of reading block (in minutes) 


101* 


92 


102* 


93 


-1.2 


Grades 1-3 

...Schools with reading block 


100%* 


92% 


100%* 


94% 


-1.6% 


...Schools with a reading block < 90 minutes 


1% 


23%* 


0% 


20%* 


2.2% 


...Length of reading block (in minutes) 


110* 


98 


111* 


97 


1.4 



Exhibit reads: In 2004-05, 99 percent of RE schools reported that their school scheduled a reading block for kindergarten, 
compared to 88 percent of non-RF Title I schools. This difference is statistically significant (p < .05). In 2006-07, 99 percent 
of RF schools reported scheduling a kindergarten reading block, compared to 91 percent of non-RF Title I schools. This 
difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by -2.6 percentage points (right-hand 
column) reflecting an 8 percentage point difference between RF and non-RF Title I schools in 2006-07 and an 1 1 percentage 
point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.3.1 for additional statistics. 

Source: Reading Coach Survey and Principal Survey (Question D2 in 2004-05 and D1 in 2006-07). 

Weight: School. 

Weighted respondents: Principals or reading coaches in 1,678 RF schools and 14,316 non-RF Title I schools. 

Non-response rates across survey items, years, and respondents: 0 to 3.6 percent. 



Teachers in RF schools reported spending significantly more time on daily reading instruction 
than teachers in non-RF Title I schools. Grade 1-3 teachers in RF schools reported spending an 
average of 104 minutes per day on reading instruction compared to 83 minutes in non-RF Title I 
schools; this translates to an additional 21 minutes per day of reading instruction for students in 
grades 1-3 in RF schools. A similar pattern was found during the 2004-05 school year (Exhibit 
3.2). 
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Exhibit 3.2 



Average Minutes Per Day Teachers Reported Devoting to Reading Instruction in RF and Non-RF 
Title I Schools, 2004-05 and 2006-07 School Years 







2004-05 


2006-07 


2006-07 




Average Minutes Per Day of Reading instruction 


Difference- 


Grade Level 


RF 


Non-RF 
Titie i 


Non-RF 
RF Titie i 


2004-05 

Difference 


Kindergarten 


99* 


74 


101* 77 


0.5 


Grades 1—3 


102* 


82 


104* 83 


1.6 



Exhibit reads: In 2004-05, teachers in RF schools reported devoting an average of 99 minutes per day on reading instruction, 
compared to 74 minutes reported by teachers in non-RF Title I schools. This difference is statistically significant (p < .05). In 
2006-07, an average of 101 minutes per day was reported by teachers in RF schools, compared to an average of 77 minutes 
reported by teachers in non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 
results differ by -0.5 percentage points (right-hand column) reflecting the fact that there was a 25 percentage point difference 
between RF and non-RF Title I schools in 2006-07 and a 24 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and 
non-RF Title I schools. 

See Appendix B, Exhibit B.3.2 for additional statistics. 

Source: Teacher Survey (Question Cl in 2004-05 and B1 in 2006-07). 

Weight: Teacher. 

Weighted respondents: Teachers in 1,649 RF schools; 13,230 non-RF Title I schools. 

Non-response rate: 2.1 to 11.3 percent. 



Instructional Materials 

The Reading First program guidance outlines strategies for states and districts to use in selecting 
or developing instructional materials, programs, learning systems and strategies to implement 
methods that have been proven effective in teaching reading skills. Substantial changes to 
schools’ reading programs may be necessary to implement these activities. 

Changing Materials 

Survey responses from principals and reading coaches indicate that RF schools made substantial 
changes in their instructional material in 2004-05, the first school year after they received their 
RF funds. By 2006-07, however, RF schools were less likely than non-RF Title I schools to 
make additional changes; 3 percent of RF schools reported adopting a new core reading program 
compared to 39 percent in 2004-05,^^ and 14 percent added new materials for ELLs, compared 
to 43 percent in 2004-05. Nonetheless, over 40 percent of both groups of schools reported 
adding new intervention programs for struggling readers and adding new supplementary 
materials during the 2006-07 school year (Exhibit 3.3). 



U.S. Department of Edueation, Offiee of Elementary and Secondary Education, Guidance for the Reading First 
Program. Washington, D.C., April 2002, page 1. 

There were no significant differences in the proportion of RF schools adopting a new core reading program in 
2006-07 if they had not done so in 2004-05. Four percent of the RF schools that had not adopted a new 
reading program in 2004-05 reported doing so in 2006-07; 2 percent of the RF schools that reported adopting a 
new program in 2004-05 also reported adopting a new program in 2006-07. 
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Exhibit 3.3 



Changes to Reading Program Materiais in RF and Non-RF Titie i Schoois, 2004-05 and 2006-07 
Schooi Years 



100 



2004-05 



*39 



*74 



43 



*69 



58 



*43 



16 



29 



— I — 

80 



— I — 

60 



— I — 

40 



— I — 

20 



Percent of Schools 



Difference in Difference+ 



Adopted new core 
reading program 

- 37 . 6 * 

Added a new intervention 
program for struggling readers 

- 37 . 2 * 

Added new supplementary 
materials 

- 26 . 8 * 



Added new materials for ELLs 
- 26 . 3 * 



1 ; 



2006-07 



□ RF Schools 

□ Non-RF Title I Schools 






40 



46 

42 



59* 



14 



26* 



20 40 60 80 100 

Percent of Schools 



Exhibit reads: In 2004-05, 39 percent of RE schools reported that their schools adopted new core reading programs, compared to 
16 percent of non-RF Title I schools. This difference is statistically significant (p < .05). In 2006-07, 3 percent of RF schools 
reported adopting a new core reading program, compared to 17 percent of non-RF Title I schools. This difference is statistically 
significant (p < .05). The 2004-05 and 2006-07 results differ by -37.6 percentage points (middle column) reflecting the fact that 
there was a 14 percentage point difference between RF and non-RF Title I schools in 2006-07 and a 23 percentage point difference 
in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

+ The difference in difference model is explained on page C-6. 

See Appendix B, Exhibit B.3.3 for additional statistics. 

Source: Reading Coach Survey (Question C3 in 2004-05 and C2 in 2006-07); Principal Survey (Question D7 2004-05 and D in 
2006-07). 

Weight: School. 

Weighted respondents: Principals or reading coaches in 1,678 RF schools and 14,341 non-RF Title I schools. 

Non-response rates across survey items: 0.1 to 5.4 percent. 



In 2006-07, non-RF Title I schools were more likely than RF schools to report that they adopted 
a new core reading program (17 percent vs. 3 percent) and that they adopted new materials for 
ELLs (26 percent vs. 14 percent). Further, in both years, nearly 60 percent of non-RF Title I 
schools reported adding new supplementary materials. 

Interestingly, when examining the same schools at two time points, a sizable number of RF and 
non-RF Title I schools added new intervention programs for struggling readers both in 2004-05 
and again in 2006-07 (30 percent and 23 percent, respectively). Similarly, 33 percent of RF 
schools and 38 percent of non-RF Title I schools added new supplementary materials to their 
reading programs in both years. 
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Quality of Materials 

RF reading coaches were significantly more likely than coaches in non-RF Title I schools to 
report that their K-3 classrooms have ample, high quality instructional materials (92 percent vs. 
72 percent) and that their reading intervention materials are aligned with scientifically based 
reading research (94 percent vs. 79 percent). This difference was found in both 2004-05 and 
2006-07 (Exhibit 3.4). 



Exhibit 3.4 

Characteristics of Reading Materials as Reported by Reading Coaches in RF and Non-RF Title I 
Schools, 2004-05 and 2006-07 School Years 

2004-05 2006-07 



*87 



73 



*87 



78 



100 



— I — 

80 



34 



*43 



— I — 

60 



— I — 

40 



— I — 

20 



Difference in Difference 



K-3 classrooms have ample, 
high quality instructional 
materials 
6.2 



K-3 teachers are 
experienced with reading 
intervention materials and 
strategies 
18.3* 



Reading intervention materials 
are aligned with SBRR 
5.2 



□RF Schools 
□Non-RF Title I Schools 



72 



er 



— I — 

20 



— I — 

40 



52 

— I — 

60 



— I — 

80 



92* 



94* 



79 



100 



Percent of Schools Percent of Schools 

Exhibit reads: In 2004-05, 87 percent of RE reading coaches reported that K-3 classrooms in their schools have ample, high 
quality instructional materials, compared to 73 percent of reading coaches in non-RF Title I schools. This difference is statistically 
significant (p < .05). In 2006-07, 92 percent of RF reading coaches reported that this statement “accurately” or “very accurately” 
described their schools, compared to 72 percent of reading coaches in non-RF Title I schools. This difference is statistically 
significant (p < .05). The 2004-05 and 2006-07 results differ by 6.2 percentage points (right-hand column) reflecting the fact that 
there was a 20 percentage point difference between RF and non-RF Title I schools in 2006-07 and a 14 percentage point difference 
in 2004-05. 

Notes The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools 

Percentages are based on respondents rating the statements as either “accurate” or “very accurate” descriptions of their schools. 

See Appendix B, Exhibit B.3.4 for additional statistics. 

Source: Reading Coach Survey (Question HI in 2004-05 and G1 in 2006-07). 

Weight: Reading coach. 

Weighted respondents: Reading coaches in 1,635 in RF schools and 7,667 non-RF Title I schools. 

Non-response rate across survey items: < 1 percent. 
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Responsibility for Selection of Materials 



Through the establishment of state Reading Leadership Teams, the Reading First Program 
Guidance encourages the involvement of a wide array of state, district and school-level staff in 
key reading policy decisions in schools. These teams monitor and examine reading instruction 
strategies in schools and approve reading instruction plans submitted by school districts applying 
for RF funding. The Program Guidance also encourages the use of reading coaches as a means 
for ongoing, continuous professional development."^*' RF schools were much more likely than 
non-RF Title I schools, to report that state staff were involved in key reading policy decisions. 
About one-quarter of the RF schools reported state involvement in the selection of their schools’ 
core reading programs, supplemental reading program materials and materials for use with 
struggling readers; in comparison, 17 percent of non-RF Title I schools reported that state staff 
were involved in the selection of a core reading program and 6-7 percent reported state 
involvement in the selection of supplemental reading materials and materials for struggling 
readers (Exhibit 3.5). 

Districts played an important role in the reading policy decisions of both RF and non-RF Title I 
schools. Despite a small, yet statistically significant, difference in the level of district 
involvement in the selection of supplemental reading materials, overall, use of district-level staff 
in key reading policy decisions was similar amongst RF and non-RF Title I schools. Almost 90 
percent of RF and non-RF Title I principals reported district involvement in the selection of their 
school’s core reading program (87 percent vs. 88 percent),"” and over half reported district 
involvement in the selection of supplemental reading program materials (66 percent vs. 56 
percent) and the selection of intervention reading program materials for use with struggling 
readers (64 percent vs. 58 percent) (Exhibit 3.5). 

RF schools were also more likely than non-RF Title I schools to involve reading coaches in key 
reading policy decisions. More than half (57 percent) of non-RF Title I schools employ reading 
coaches (compared to 99 percent of RF schools); those that did were significantly less likely than 
RF schools to report that reading coaches had responsibility for selecting a core reading program 
(29 percent vs. 41 percent) or supplemental reading program materials (60 percent vs. 70 
percent) (Exhibit 3.5). 



U.S. Department of Edueation, Offiee of Elementary and Secondary Education, Guidance for the Reading First 
Program. Washington, D.C., April 2002, pages 15-16. 

U.S. Department of Education, Office of Elementary and Secondary Education, Guidance for the Reading First 
Program. Washington, D.C., April 2002, page 26. 

Fifty-three percent of the RF schools and 59 percent of the non-RF Title I schools reported that responsibility 
for selecting a specific core reading program rested solely with the district or state. For these schools, principals 
and reading coaches had no responsibility for selecting their school’s core reading program. 
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Exhibit 3.5 



Responsibility for Selection of Reading Materials in RF and Non-RF Title I Schools, 2004-05 and 2006-07 
School Years 



85 

87 



Specific Core Reading Program 
2004-05 2006-07 



*31 



Difference in Difference 
30* 



19 



*46 




40 




- 


*37 




29 



State 

0.4 

District 

1.3 



Principal 

1.2 



Reading 
Coach ^ 
3.0 



100 80 60 40 20 0 

Percent of Schools 



17 



45* 

39 



41* 



29 



87 

88 



— I 1 1 1 1 

20 40 60 80 100 
Percent of Schools 



Supplemental Reading Materials 
2004-05 2006-07 

Difference in Difference 




100 80 60 40 20 C 

Percent of Schools 



20 40 60 80 100 
Percent of Schools 



Intervention Reading Program Materials for Use with Struggling Readers 
2004-05 



64 

60 



68 

*74 



74 

69 

— I — 



Difference in Difference 



2006-07 



*22 




State 




25* 


□ RF Schoois 




5 | 


3.1 


I® 




■ Non-RF Titie i Schoois 



District 

1.7 

Principal 

1.6 



Reading 
Coach ^ 



-0.9 



64 

58 



68 

73 



72 

68 



100 80 60 40 20 0 

Percent of Schools 



0 20 40 60 80 100 

Percent of Schools 



Exhibit reads: In 2004-05, 31 percent of RE principals reported that state staff were involved in the selection of specific core reading 
programs in their schools, compared to 19 percent of principals in non-RF Title I schools. This difference is statistically significant (p < 
.05). In 2006-07, 30 percent of RF principals reported that state staff were involved in this selection in their schools, compared to 17 
percent of principals in non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ 
by 0.4 percentage points (middle column) reflecting the fact that there was a 13 percentage point difference between RF and non-RF Title I 
schools in 2006-07 and a 12 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I schools. 

See Appendix B, Exhibit B.3.5 for additional statistics. 

^ Analysis of reading coach responsibilities is limited to schools with a reading coach. 

Source: Principal Survey (Question D4 in 2004-05 and D2 in 2006-07). 

Weight: Principal. 

Weighted respondents: Principals in 1,678 RF schools and 14,602 non-RF Title I schools. 

Non-response rate across survey items: 0 to 7.6 percent. 
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Corroborating principal reports, reading coaches indicated that there is no significant difference 
between RF and non-RF Title I schools in the likelihood that the district provides direction 
concerning reading instruction (72 percent vs. 66 percent). Reading coaches also indicated that 
states are more likely to provide direction in RF schools than in non-RF Title I schools (80 
percent vs. 61 percent) (Exhibit 3.6). However, it is interesting to note that the percentage of 
both RF and non-RF school reading coaches reporting that the state provides direction 
concerning reading instruction is much higher than the percentage of principals indicating that 
the state was actually involved in key policy decisions. 



Exhibit 3.6 

State and District Guidance in RF and Non-RF Titie i Schoois as Reported by Reading Coaches, 
2004-05 and 2006-07 School Years 



2004-05 2006-07 




Percent of Schools Percent of Schools 

Exhibit reads: In 2004-05, 73 percent of RE reading coaches reported that the district provides direction concerning reading 
instruction in their schools, compared to 70 percent of reading coaches in non-RF Title I schools. This difference is not statistically 
significant. In 2006-07, 72 percent of RF reading coaches reported that this statement “accurately” or “very accurately” described 
their schools, compared to 66 percent of reading coaches in non-RF Title I schools. This difference is not statistically significant. 
The 2004-05 and 2006-07 results differ by 2.6 percentage points (right-hand column) reflecting the fact that there was a 6 
percentage point difference between RF and non-RF Title I schools in 2006-07 and a 3 percentage point difference in 2004-05. 
Notes The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

Percentages are based on respondents rating the statements as either “accurate” or “very accurate” descriptions of their schools. 

See Appendix B, Exhibit B.3.6 for additional statistics. 

Source: Reading Coach Survey (Question HI in 2004-05 and G1 in 2006-07). 

Weight: Reading coach. 

Weighted respondents: Reading coaches in 1,626 RF schools and 7,590 non-RF Title I schools. 

Non-response rate across survey items: 0 to 3.6 percent. 
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Core Reading Programs 

According to the Reading First Program Guidance, the centerpiece of instruction should be a 
core reading program aligned with SBRR that provides instruction in the five dimensions of 
reading."^^ RF schools were significantly more likely to report using a single core reading 
program for all of their K-3 classrooms than were non-RF Title 1 schools (78 percent vs. 58 
percent) (Exhibit 3.7). Non-RF Title I schools were significantly more likely to report multiple 
core programs, either within or across grade levels than were RF schools (42 percent vs. 22 
percent). 

The use of multiple programs may reflect differentiated instructional needs (i.e., one program for 
grades K-1, and another for grades 2-3, or one program for native English speaking students and 
one for ELLs). However, schools using multiple programs presumably must train teachers on 
more than one reading program, an experience which may lead to increased demands on coaches 
and principals to monitor their implementation, and perhaps less continuity of instruction across 
grades or subgroups of students. 



Exhibit 3.7 



Number of Core Reading Programs Used in Grades K— 3 in RF and Non-RF Titie i Schoois, 2006- 
07 Schooi Year 



Number of Core Reading Programs 


RF Schools 


Non-RF Title I Schools 


One core reading program for each grade served, K-3 






Same program across grades 


78%* 


58% 


Multiple programs across grades K-3 
Multiple core reading programs within at least one 


3 


9* 


grade served, K— 3 


19 


33* 



Exhibit reads: In 2006-07, 78 percent RF schools reported using a single core reading program for all their K-3 classrooms, 
compared to 58 percent of non-RF Title I schools,. This difference is statistically significant (p < .05). 

Notes: Fimited to schools reporting at least one reading program for the specified grade or across grades. 

The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B. Exhibit B.3.7 for additional statistics. 

Source: Reading Coach Survey (Question Cl in 2006-07); Principal Survey (Question Cl in 2006-07). 

Weight: School. 

Weighted respondents: Principals or reading coaches in 1,551 RF schools and 1 1,768 non-RF Title I schools. 

Non-response rate: 0.2 to 6.9 percent. 



U.S. Department of Education, Office of Elementary and Secondary Education, Guidance for the Reading First 
Program. Washington, D.C., April 2002, page 6. 
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Virtually all RF and non-RF Title I schools implemented at least one core reading program in 
each of grades K-3."^^ In 2006-07, a total of 48 different core reading programs were identified 
in RF schools and 68 in non-RF Title I schools. Although these counts are higher than those 
reported identified in 2004-05,"^"^ the list of most commonly used reading programs remained 
similar. 

The most frequently cited core reading programs in RF schools in 2006-07 included Harcourt 
Trophies (23 percent), McGraw-Hill Open Court (15 percent), Scott Foresman Reading (13 
percent) and Houghton Mifflin Nation’s Choice (1 1 percent); with the exception of the Scott 
Foresman Reading Program, all of these programs ranked among the top five programs cited in 
200^05 (Exhibit 3.8). 

Four of the five reading programs cited most frequently by RF schools were also among the five 
most popular programs in the non-RF Title I schools. At least 10 percent of both RF and non-RF 
Title I schools reported using Harcourt Trophies, McGraw-Hill Open Court, Scott Foresman 
Reading and Houghton Mifflin Reading. The exceptions were Houghton Mifflin Nation’s 
Choice and McGraw-Hill Reading. Houghton Mifflin Nation’s Choice was used by almost 1 1 
percent of RF schools, but less than 3 percent of non-RF Title I schools. McGraw-Hill Reading 
was used by 1 1 percent of non-RF Title I schools, but only 7 percent of RF schools. 



Only 3 percent of non-RF Title I schools reported that they did not have a core reading program for at least one 
of grades K-3; all RF schools had core reading programs for grades K-3. In 2 percent of RF schools and 7 
percent of non-RF Title I schools, staff reported using a program developed by teachers or other school 
persoimel for one or more of their K-3 classrooms. 

It is important to note that the structure of this question was revised between the 2004-05 and 2006-07 survey. 
In 2004-05, principals and reading coaches were asked to give open-ended responses to questions about core 
reading programs used in grades K-3. In 2006-07, a number of close-ended response options were added based 
on high frequency responses to the 2004-05 survey. This resulted in a decrease in the number of “unspecified” 
responses. Moreover, the non-response rate among non-RF Title I schools dropped from 19 percent in 2004-05 
to 4 percent in 2006-07. Therefore, this may explain the increase in diversity in core reading programs listed in 
2006-07. 

The Scott Foresman Reading program was reported by less than 6 percent of RF schools in 2004-05. 
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Exhibit 3.8 



Core Reading Programs Used by RF and Non-RF Title I Schools as Reported by Principals and 



Reading Coaches, 2006-07 School Year. 



Publisher 


Program 


Percentage of Schools ^ 


RF 


Non-RF Title 1 


Harcourt 


Collections 


0.9 


7.6 




Rigby Reading 


2.1 


4.2 




Signatures 


0.0 


1.0 




Trophies 


22.5 


16.0 


Heinemann 


Fountas Pinnel units of study 


0.0 


5.3 


Houghton Mifflin 


Horizons 


1.6 


3.1 




Invitation to Literacy 


1.6 


4.6 




Lectura (Spanish Version of Reading California) 


5.7 


2.3 




Legacy of Literacy 


4.2 


4.8 




Nation's Choice 


10.7 


2.5 




Reading 


9.8 


10.1 




State Specific Edition ^ 


3.3 


2.3 


McGraw-Hill 


Lectura 


1.0 


0.0 




Open Court 


15.4 


9.8 




Reading 


7.0 


10.5 




Reading Mastery 


5.8 


4.1 




Spotlight on Literacy 


0.0 


2.0 




Treasures/Triumphs 


0.0 


1.7 


Saxon 


Saxon Phonics 


0.2 


6.8 


Scholastic 


Literacy Place 


0.1 


2.9 


Scott Foresman 


Literacy Works 


0.3 


3.0 




Reading 


13.0 


12.2 




State Specific Edition ^ 


2.4 


4.9 


Sophs 


Read Well 


2.8 


1.7 


Success for All 


Success for All 


3.2 


4.1 


Voyager 


Universal Literacy 


4.8 


2.2 


Wright Group 


Unspecified 


0.6 


5.4 


Other (Unspecified) 


1.6 


4.9 


Core reading program developed by teachers or other school personnel 


1.5 


7.3 



Exhibit reads: In 2006-07, 22.5 percent of RF schools (reading coaches or principals) reported using the “Trophies” reading 



program published by Harcourt for at least one of grades K-3. 

See Appendix B, Exhibit B.3.8 for additional statistics. 

^ Limited to schools reporting at least one reading program for their K-3 classrooms. The denominator excludes the 2.2 percent 
of non-RF Title I schools that did not use at least one reading program for their K-3 classrooms. None of the RF schools 
reported that they did not use a core reading program for these grades. 

^ A number of publishers have developed programs tailored specifically to the needs of individual states, referred to here as 
“state specific editions.” 

Source: Reading Coach Survey (Question Cl in 2006-07); Principal Survey (Question Cl in 2006-07). 

Weight: School. 

Weighted respondents: Principals or reading coaches in 1,552 RF schools and 12,086 non-RF Title I schools. 

Non-response rate: 0.1 to 6.4 percent. 



According to reading coaches, RF schools’ core reading programs are more likely to be aligned 
with SBRR than are non-RF Title I schools’ reading programs (93 percent vs. 76 percent). In 
2006-07, reading coaches in RF schools were also significantly more likely than coaches in non- 
RF Title I schools to report that their K-3 teachers are experienced with this core reading 
program (88 percent vs. 69 percent). This represents a substantial increase from the 63 percent 
reported by coaches in 2004-05 (Exhibit 3.9). There was no meaningful change in reading 
coaches’ ratings over this time period in non-RF Title I schools (72 percent and 69 percent, 
respectively). 
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Exhibit 3.9 



Reading Coach Reports of Teacher Experience with Core Reading Program in RF and Non-RF Titie I 
Schoois, 2004-05 and 2006-07 Schooi Years 



2004-05 



2006-07 



Difference in Difference 



□ RF Schools 

□ Non-RF Title I Schools 










76 





00 

00 




69 
















0 


1 

20 


1 

40 


1 

60 


1 

80 


1 

100 



Percent of Schools Percent of Schools 

Exhibit reads: In 2004-05, 94 percent of RF reading coaches reported that the core reading program used in their schools is aligned 
with SBRR, compared to 78 percent of reading coaches in non-RF Title I schools. This difference is statistically significant (p < 
.05). In 2006-07, 93 percent of RF reading coaches reported that this statement “accurately” or “very accurately” described their 
school, compared to 76 percent of reading coaches in non-RF Title I schools. This difference is statistically significant (p < .05). 
The 2004-05 and 2006-07 results differ by 1.2 percentage points (right-hand column) reflecting the fact that there was a 17 
percentage point difference between RF and non-RF Title I schools in 2006-07 and a 16-percentage point difference in 2004-05. 
Notes The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

Percentages are based on respondents rating the statements as either “accurate” or “very accurate” descriptions of their schools. 

See Appendix B, Exhibit B.3.9 for additional statistics. 

Source: Reading Coach Survey (Question in 2004-05 and G1 in 2006-07). 

Weight: Reading coach. 

Weighted respondents: Reading coaches in 1,626 RF schools and 7,590 non-RF Title I schools. 

Non-response rate across survey items: 0 to 3.6 percent. 



Supplemental Reading Materials 

Supplementary reading materials provide additional instruction, as needed, in a targeted area 
(e.g., phonemic awareness, fluency, vocabulary or comprehension). The use of supplemental 
materials to support instruction in the five key elements of reading instruction is an important 
component of Reading First, as is the expectation that these materials be aligned with SBRR. 
Teachers have become more experienced users of supplemental materials according to reading 
coaches; 67 percent of RF coaches and 56 percent of coaches in non-RF Title I schools reported 
that K-3 teachers are experienced with supplemental reading materials (Exhibit 3.10). This 
represents a significant change for RF teachers from 2004-05 when only 38 percent of reading 
coaches in RF schools rated their teachers as experienced in using supplemental materials. 
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Exhibit 3.10 



Suppiementai Reading Materiais in RF and Non-RF Titie I Schoois as Reported by Reading 
Coaches, 2004-05 and 2006-07 Schooi Years 

2004-05 2006-07 




Percent of Schools Percent of Schools 

Exhibit reads: In 2004-05, 81 percent of RE reading coaches reported that the supplemental reading materials used in their schools 
are aligned with SBRR, compared to 71 percent of reading coaches in non-RF Title I schools. This difference is statistically 
significant (p < .05). In 2006-07, 89 percent of RF reading coaches reported that this statement “accurately” or “very accurately” 
described their school, compared to 74 percent of reading coaches in non-RF Title I schools. This difference is statistically 
significant (p < .05). The 2004-05 and 2006-07 results differ by 5 percentage points (right-hand column) reflecting the fact that 
there was a 15 percentage point difference between RF and non-RF Title I schools in 2006-07 and a 10 percentage point difference 
in 2004-05. 

Notes The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

Percentages are based on respondents rating the statements as either “accurate” or “very accurate” descriptions of their schools. 

See Appendix B, Exhibit B.3.10 for additional statistics. 

Source: Reading Coach Survey (Question HI in 2004-05 and G1 in 2006-07). 

Weight: Reading coach. 

Weighted respondents: Reading coaches in 1,626 RF schools and 7,590 non-RF Title I schools. 

Non-response rate across survey items: 0 to 3.6 percent. 



Additionally, RF teachers were more likely to report using materials that supplement the core 
reading program with struggling readers than were teachers in non-RF Title I schools (83 percent 
vs. 71 percent). 

Instructional Activities and Strategies 

The Reading First program, at its core, is designed to align teachers’ instruction with SBRR. In 
this section, we describe the characteristics of reading instruction, staff collaboration and 
instructional activities teachers engage in to teach reading. 
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Teacher Knowledge of SBRR 

To be able to align their instruction with SBRR, K-3 teachers must be knowledgeable about 
SBRR. More reading coaches in RF schools reported that their K-3 teachers in RF schools are 
knowledgeable about SBRR than did coaches in non-RF Title I schools (79 vs. 58 percent) 
(Exhibit 3.1 1). 



Exhibit 3.11 



Knowledge and Motivation of Teachers in RF and Non-RF Title I Schools as Reported by Reading 
Coaches, 2004-05 and 2006-07 School Years 

2004-05 2006-07 




Percent of Schools Percent of Schools 

Exhibit reads: In 2004-05, 57 percent of RE reading coaches reported that K-3 teachers in their schools are knowledgeable about 
scientifically based reading instruction, compared to 48 percent of reading coaches in non-RF Title I schools. This difference is 
statistically significant (p < .05). In 2006-07, 79 percent of RF reading coaches reported that this statement “accurately” or “very 
accurately” described their school, compared to 58 percent of reading coaches in non-RF Title I schools. This difference is 
statistically significant (p < .05). The 2004-05 and 2006-07 results differ by 1 1.6 percentage points (right-hand column) reflecting 
the fact that there was a 21 percentage point difference between RF and non-RF Title I schools in 2006-07 and a 9 percentage point 
difference in 2004-05. 

Notes The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

Percentages are based on respondents rating the statements as “accurate” or “very accurate” descriptions of their schools. 

See Appendix B, Exhibit B.3.1 1 for additional statistics. 

Source: Reading Coach Survey (Question HI in 2004-05 and G1 in 2006-07). 

Weight: Reading coach. 

Weighted respondents: Reading coaches in 1,626 schools and 7,590 non-RF Title I schools. 

Non-response rate across survey items: 0 to 3.6 percent. 
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Collaboration on Reading Instruction 



According to teacher reports, RF schools were more likely than non-RF Title I schools to 
formally set aside at least some time for collaboration, observation and teacher-to-teacher 
coaching on reading instruction. For instance, teachers in non-RF Title I schools were somewhat 
more likely to report that no time was set aside for collaboration on reading lesson planning and 
instruction (17 percent vs. 10 percent), for observation of reading instruction in other classrooms 
(64 percent vs. 53 percent) and for teachers to help with coaching or be coached about reading by 
another teacher (40 percent vs. 22 percent) (Exhibit 3.12). 



Exhibit 3.12 



Type and Frequency of Collaboration about Reading in RF and Non-RF Title I Schools, 2004-05 and 
2006-07 School Years 





2004- 


-05 


2006- 


-07 


2006-07 


Type of Collaboration/ 
Frequency of Time Set Aside to: 


Teachers 


Teachers 


Difference- 


RF 


Non-RF 
Title 1 


RF 


Non-RF 
Title 1 


2004-05 

Difference 


Collaborate on reading lesson planning and 
Instruction 












...Not at all 


13% 


20%* 


10% 


17%* 


-0.1% 


...Monthly or less 


35* 


32 


40* 


32 


3.8 


. . .Once a week or more 


38* 


32 


42* 


38 


-2.2 


...Informally, as needed 


13 


16 


8 


13* 


-1.6 


Observe reading Instruction In other classrooms 












...Not at all 


59 


65* 


53 


64* 


-5.0* 


...Monthly or less 


20 


18 


23* 


16 


4.7* 


. . .Once a week or more 


1* 


1 


2 


2 


-0.8 


...Informally, as needed 


19* 


17 


23* 


19 


1.0 


Help with coaching or be coached about reading 
by other teacher 












...Not at all 


25 


41* 


22 


40* 


-2.1 


...Monthly or less 


40* 


34 


46* 


33 


6.2* 


. . .Once a week or more 


15* 


8 


14* 


8 


-2.3 


...Informally, as needed 


19 


18 


18 


19 


-1.8 



Exhibit reads: In 2004-05, 13 percent of teachers in RE schools reported that they do not collaborate on reading lesson planning 
and instruction, compared to 20 percent of teachers in non-RF Title I schools. This difference is significantly significant (p < .05). 
In 2006-07, 10 percent of teachers in RF schools reported no collaboration on this activity compared to 17 percent in non-RF 
Title I schools. This difference is significantly significant (p < .05). The 2004-05 and 2006-07 results differ by -0.1 (right-hand 
column) reflecting the fact that there was a 7 percentage point difference between RF and non-RF Title I schools in 2006-07 and a 
7 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.3.12 for additional statistics. 

Source: Teacher Surveys (Question C3 in 2004-05 and B3 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers 1,682 RF and 14,656 non-RF Title I schools. 

Non-response rate across survey items: < 1 percent. 
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Instructional Activities 

To ensure that children learn to read well, the RF Guidance recommends that explicit and 
systematic instruction be provided in five essential areas identified by SBRR as necessary for 
effective reading instruction."^^ The surveys asked teachers to describe how central individual 
instructional activities are to their teaching, and in order to describe adherence to broader RF 
principles, we constructed six composite measures from the individual survey items (See 
Appendix E for details on individual items in each composite). The composites summarize 
teachers’ ratings of instructional activities related to the following reading dimensions and other 
instructional features: 

• phonemic awareness and phonics; 

• comprehension; 

• vocabulary development; 

• reading fluency; 

• use of scientifically based instructional strategies and materials in their classroom 
(SBRR); and 

• use of instructional strategies and materials that depart from scientifically based reading 
research (non-SBRR)."^’ 

Composite scores were based on the percentage of instructional activities rated as central to 
teachers’ instruction (Exhibit 3.13). For example, if a teacher rated six of the seven activities in 
the comprehension composite as central, the composite “score” would be 86 percent. 

K-3 teachers in RF schools rated a higher proportion of scientifically based teaching strategies 
and materials (the shaded rows in Exhibit 3.13) as central to their instruction than did teachers in 
non-RF Title I schools. The few statistically significant differences observed across the other 
composites were not substantively meaningful. No differences were found between teachers in 
RF and non-RF Title I schools on their ratings of the centrality of non- SBRR. instructional 
activities. 



U.S. Department of Education, Office of Elementary and Secondary Education, Guidance for the Reading First 
Program. Washington, D.C., April 2002, page 3. 

The phonemic awareness and phonics composite, for example, includes the following student activities; 
isolating first, middle and ending sounds in words, matching sounds and letters, and blending sounds to form 
words. The SBRR composite is based on teachers’ responses to all items included in four composites (phonemic 
awareness and phonics, comprehension, vocabulary, and fluency) presented in Exhibit 3.13. The non-SBRR 
composite includes several instructional activities for which there is no scientifically based evidence of 
effectiveness, including, for example, students reading silently, memorizing sight words, reading unfamiliar 
texts aloud, and reading texts that are easy to decode. 
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Exhibit 3.13 



Centrality of SBRR-aligned Instructional Activities in RF and Non-RF Title I Schools, 2004-05 and 
2006-07 School Years 





2004- 


-05 


2006- 


-07 






Mean Percent of Teachers’ SBRR-aiigned 
instructionai Activities 


2006-07 

Difference- 




RF 


Non-RF 
Titie i 


RF 


Non-RF 
Titie i 


2004-05 

Difference 


Phonemic Awareness and Phonics 












Kindergarten 


90%* 


85% 


92%* 


87% 


0.0% 


1st grade 


86* 


81 


88* 


83 


0.1 


2nd grade 


65* 


60 


68* 


59 


3.6 


3rd grade 


58 


54 


60* 


49 


7.0* 


Comprehension 












Kindergarten 


72* 


67 


73* 


68 


-0.6 


1st grade 


75* 


71 


76 


73 


-1.1 


2nd grade 


69 


69 


67 


68 


-0.5 


3rd grade 


73* 


69 


74* 


69 


1.8 


Vocabuiary 












Kindergarten 


64* 


59 


63* 


59 


-1.1 


1st grade 


87 


87 


90 


88 


2.4 


2nd grade 


72 


72 


71 


73 


-1.2 


3rd grade 


79* 


74 


79 


76 


-1.4 


Fiuency 

Kindergarten 












1st grade 


86 


84 


87* 


84 


2.3 


2nd grade 


56 


59 


59 


56 


5.6* 


3rd grade 


57* 


46 


57* 


47 


-1.5 


Overaii Composite SBRR 












Kindergarten 


79* 


66 


85* 


71 


1.5 


1st grade 


82* 


76 


88* 


79 


2.4 


2nd grade 


79* 


73 


84* 


77 


-0.1 


3rd grade 


78* 


69 


82* 


73 


-0.7 


Overaii Composite Non-SBRR 












Kindergarten 


67 


65 


65 


65 


-2.3 


1st grade 


69 


71 


70 


71 


1.7 


2nd grade 


67 


69* 


67 


67 


1.9 


3rd grade 


65 


64 


64 


64 


-1.1 



Exhibit reads: In 2004-05, kindergarten teaehers in RE sehools rated an average of 90 pereent of SBRR-aligned phonemie 
awareness and phonies aetivities as eentral to their instruetion, eompared to an average of 85 pereent reported by teaehers in non- 
RF Title I sehools. This differenee is statistieally signifieant (p < .05). In 2006-07, kindergarten teaehers in RF sehools rated an 
average of 92 pereent of SBRR-aligned phonemie awareness and phonies aetivities as eentral to their instruetion, eompared to an 
average of 87 pereent reported by teaehers in non-RF Title I sehools. This differenee is statistieally signifieant (p < .05). The 
2004-05 and 2006-07 results differ by 0 pereentage points (right-hand eolumn) refleeting the faet that there was a 5 pereentage 
point differenee between teaehers in RF and non-RF Title I sehools in 2006-07 and a 5 pereentage point differenee in 2004-05. 
Notes: The stars (*) indieate that there is a statistieally signifieant differenee (i.e., p-value < .05) between teaehers in RF and 
non-RF Title I sehools. 

Kindergarten teaehers were not asked about flueney instruetion. 

See Appendix B, Exhibit B.3.13 for additional statisties and Appendix D for the speeifie items ineluded in eaeh eomposite at 
eaeh grade level. 

Souree: Teaeher Surveys (Question C4 in 2004-05 and B4 in 2006-07). 

Weight: Teaeher. 

Weighted respondents: Teaehers in 1,635 RF and 13,756 non-RF Title I sehools. 

Non-response rate aeross survey items: 0.3 to 3.5 pereent. 
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Summary 



RF schools consistently reported higher alignment with RF principles than non-RF Title I 
schools on each component of a reading program; instructional time, materials, knowledge of 
and experience with SBRR, and collaboration. For the most part, significant differences 
observed in the first school year after initial receipt of RF funds (2004-05) and continued 
through subsequent years (2006-07). At the same time, reading instruction in non-RF Title I 
schools is becoming increasingly aligned with the principles of Reading First. 

A majority of RF schools reported making substantive structural changes to their reading 
programs either in 2004-05. Many changes, including the adoption of a new core reading 
program, were made only once, during the initial year of RF implementation. However, over 40 
percent of RF schools added new supplementary materials or added new intervention programs 
for struggling readers in 2006-07. Many non-RF Title I schools also reported making changes to 
their reading programs. In each year (2004-05 and 2006-07), almost 60 percent of these schools 
reported that they added new supplementary materials and more than 40 percent added new 
interventions for struggling readers. 

Reading First schools were more likely than non-RF Title I schools to involve a wide array of 
state, district and school-level staff in key reading policy decisions. In particular, RF schools 
were more likely to have involved state staff and school reading coaches in the selection of 
appropriate materials for their reading programs. 

Virtually all RF schools scheduled reading blocks in K-3 classrooms of 90 minutes or more. 
Although a majority of non-RF Title I schools scheduled reading blocks, both the prevalence of 
these blocks and their length were significantly less than in RF schools. Moreover, K-3 teachers 
in RF schools reported, on average, spending significantly more time on daily reading activities 
than K-3 teachers in non-RF Title I schools (103 minutes vs. 81 minutes). 

The vast majority of RF schools (99 percent) reported having at least one core reading program 
for each of grades K-3 and almost 80 percent used the same program across all of their K-3 
classrooms. The five most frequently used core reading programs in RF schools included 
Harcourt Trophies (23 percent), McGraw-Hill Open Court (15 percent), Scott Foresman Reading 
(13 percent), Houghton Mifflin Nation’s Choice (11 percent) and Houghton Mifflin’s Reading 
(10 percent). With the exception of Houghton Mifflin Nation’s Choice, all of these programs 
were also the programs most commonly cited by non-RF Title I schools. 

Consistent with the tenets of the RF program, reading instruction in RF schools was more likely 
than instruction in non-RF Title I schools to be aligned with SBRR. This finding is supported by 
both reading coach perceptions of the characteristics of reading instruction in their schools (e.g., 
whether teachers are knowledgeable about SBRR and the school’s core reading program) and 
teacher ratings on the importance of SBRR-aligned practices to their reading instruction. Despite 
these differences, coaches in three-quarters of non-RF schools indicated that their core reading 
programs were aligned with SBRR. Further, reading coaches in more non-RF Title I schools 
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were likely to rate their K-3 teachers as knowledgeable about scientifically based reading 
instruction in 2006-07 than in 2004-05 (58 percent vs. 48 percent). 
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Chapter 4: Support for Struggling Readers 

A core tenet of the Reading First program is to provide additional support to students who are 
struggling to learn to read. This is especially important for those children in grades K-3 who are 
English language learners (ELL) or who are in jeopardy of being referred for special education 
services. RE schools and teachers can offer supports necessary for these students by providing 
appropriately targeted instruction and interventions. 

This chapter presents findings on the methods and activities schools implement to help 
struggling students learn to read, including a) the use of reading interventions (i.e., programs 
specifically explicitly for struggling readers in addition, to the core reading program); b) other 
instructional supports (e.g., struggling students work with a more advanced peer, a reading 
specialist in small groups, or a one-on-one with a tutor;) and c) coordination of these two 
(interventions and supports) with other instructional activities provided for special education and 
ELL students. 

Key Findings 

Addressing the needs of students who are struggling to read is an important issue for both 
RF and non-RF Title I schools; according to principals’ reports, about 28 percent of K-3 
students in both types of schools were participating in interventions to improve their 
reading skills. RF schools’ survey responses at both time points (2004-05 and 2006-07) 
indicate continued and substantial attention to the needs of struggling readers. Survey 
information provided by non-RF Title I school staff indicates that their activities to assist 
struggling readers are increasingly similar to the activities conducted in RF schools. 

The vast majority of principals in RF and non-RF Title I schools (91 to 99 percent) 
reported using various reading test scores (e.g., diagnostic tests, tests built into the core 
reading program, progress monitoring tests) and teacher recommendations to identify 
students for reading interventions. In schools that reported having reading coaches, RF 
schools were much more likely to use reading coach recommendations than were non-RF Title I 
schools (94 percent vs. 60 percent). In contrast, non-RF Title I schools were reportedly more 
likely to use requests from parents to identify students for reading interventions than RF schools 
(76 percent vs. 66 percent). 

Materials and activities specifically aimed at helping struggling readers were available in 
most RF and non-RF Title I schools (91 percent and 85 percent, respectively). Virtually all 
RF and non-RF Title I schools engage in a variety of activities to meet the needs of struggling 
readers by providing them with additional practice opportunities, and direct instruction, as well 
as further help from trained aides or volunteers. However, RF and non-RF Title I schools differ 
significantly in the content and character of these services. RF teachers were more likely to 
report placing struggling readers into intervention services in the previous month (80 percent vs. 
63 percent) and to report using diagnostic assessments to determine struggling readers’ core 
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deficits in the previous month (84 percent vs. 61 percent) than were teachers in non-RF Title I 
schools. 

Of those schools with ELL students, the percentage of RF teachers who reported setting aside 
time to coordinate with ELL staff increased dramatically from 44 percent in 2004-05 to 71 
percent in 2006-07; a nearly identical increase was reported by teachers in non-RF Title I 
schools over the same time period (from 40 percent to 70 percent). 

Reading Intervention Services 

Information Used to Identify Students for Reading Interventions 

Principals use a variety of methods to identify students needing reading interventions. In 2006- 
07, most principals (more than 90 percent) in both RF and non-RF Title I schools reported using 
various reading test scores (e.g., diagnostic tests, tests that are part of the core reading program, 
progress monitoring tests) and teacher recommendations to identify students for reading 
interventions (Exhibit 4.1). 

There are some differences between RF and non-RF Title I schools in methods used to identify 
struggling readers for interventions. In schools that have reading coaches, a significantly greater 
percentage of principals in RF schools reported using reading coach recommendations to identify 
students for reading interventions than did principals in non-RF Title I schools (94 percent vs. 60 
percent). At the same time, a significantly smaller percentage of RF principals reported using 
parent requests to identify students for reading interventions compared to principals in non-RF 
Title I schools (66 percent vs. 76 percent). 

In general, there were no substantive changes from 2004-05 to 2006-07 in the reported use of 
methods to identify students for reading interventions. One exception was with the use of 
standardized achievement scores; from 2004-05 to 2006-07, the percent of RF school principals 
who reported using standardized scores increased (from 80 percent to 87 percent), while there 
was little change for non-RF Title I schools (from 91 percent to 89 percent). 
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Exhibit 4.1 



Methods Used to identify Students for Reading Interventions in RF and Non-RF Titie I Schoois, 2004- 
OS and 2006-07 School Years 



2004-05 



2006-07 



Difference in Difference 
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92 
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96 




*98 1 
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93 
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93 
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- 
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52 
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- 


88 
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- 
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- 


70 


*77 






- 


64 


*78 
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Progress monitoring test 
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Teacher 
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part of the reading program 
2.8 
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in reading 
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recommendation 
- 5.0 

Screening test scores 
in reading 
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Documented classroom 
observations 
- 3.7 

Standardized achievement 
scores in reading 
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Other school staff 
recommendation 
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Requests from 
parents 
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87 




89 



71 

73 



66 

~| 76* 

1 1 1 1 1 

0 20 40 60 80 100 



Percent of Schools 



Percent of Schools 



Exhibit reads: In 2004-05, 97 percent of principals in RF schools reported using progress monitoring tests to identify students for 
reading interventions, compared to 92 percent of principals in non-RF Title I schools. This difference is statistically significant (p < 
.05). In 2006-07, 99 percent of principals in RF schools reported using progress monitoring compared to 94 percent by principals in 
non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by -1.1 
percentage points reflecting that there was a 5 percentage point difference in 2006-07 and a 5 percentage point difference in 2004-05. 
Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.4.1 for additional statistics. 

Source: Principal Survey (Question E2 in 2004-05 and 2006-07). 

Weight: Principal. 

Weighted respondents: Principals in 1,684 RF schools and 14,684 non-RF Title I schools. 

Non-response rate across items, groups, and years: < 1 percent. 
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Availability of Reading Intervention Services 

Once identified as straggling readers, students can then receive intervention services. According 
to principal reports, the vast majority of RF and non-RF Title I schools offer intervention 
services, although RF schools are slightly more likely to do so (91 percent vs. 85 percent) 
(Exhibit 4.2). It is noteworthy that from 2004-05 to 2006-07, the percentage of principals 
reporting the availability of intervention services increased in both RF schools (from 87 percent 
to 91 percent) and non-RF Title I schools (from 78 percent to 85 percent). In both RF and non- 
RF Title I schools, principals reported that approximately one-third of students were 
participating in interventions to improve their reading skills. The wait time for students in need 
of such services was reported to be less than a week, on average, in both RF and non-RF Title I 
schools. 



Exhibit 4.2 



Availability of Intervention Services in RF and Non-RF Title I Schools, 2004-05 and 2006-07 School 
Years 





2004-05 


2006-07 


2006-07 
Difference 
- 2004-05 
Difference 


Non-RF 
RF Title 1 

Schools Schools 


Non-RF 
RF Title 1 

Schools Schools 


Availability of reading intervention 
services 


87%* 78% 


91 %* 85% 


-2.5% 



Exhibit reads: In 2004-05, 87 percent of principals in RF schools reported that reading intervention services were available, 
compared to 78 percent of principals in non-RF Title I schools. This difference is statistically significant (p < .05). In 2006-07, 91 
percent of principals in RF schools reported that reading intervention services were available, compared to 85 percent in non-RF 
Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by -2.5 percentage 
points (right-hand column) reflecting that there was a 9 percentage point difference in 2006-07 and a 6 percentage point difference 
in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.4.2 for additional statistics. 

Source: Principal Survey (Question E3 in 2004-05 and 2006-07). 

Weight: Principal. 

Weighted respondents: Principals in 1,645 RF schools and 14,455 non-RF Title I schools. 

Non-response rate across items, groups, and years: 0.9 to 2.4 percent. 



Use of Reading Intervention Services 

A significantly greater percentage of RF teachers than teachers in non-RF Title I schools 
reported placing their straggling readers into special intervention programs in the previous month 
(80 percent vs. 63 percent). This is an increase for teachers in both types of schools from 2004- 
05 to 2006-07 (from 73 percent to 80 percent for RF schools and from 56 percent to 63 percent 
for non-RF Title I schools). 

Supports Schools Use to Meet the Needs of Struggling Readers 

Supports as Reported by Teachers 

Once students are identified as straggling readers, teachers reported using multiple supports, 
which may or may not be part of formal reading interventions, to meet their needs, including 
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conducting diagnostic assessments to determine student core deficits, working with a more 
advanced peer, a reading specialist in small groups, or working one-on-one with a tutor. 

In 2006-07, significantly more RF teachers reported using diagnostic assessments to determine 
students’ core deficits within the previous month than did teachers in non-RF Title I schools (84 
percent vs. 67 percent) (Exhibit 4.3). Additionally, a significantly greater percentage of RF 
teachers than teachers in non-RF Title I schools reported placing struggling readers into small 
groups with a reading specialist within the previous month (59 percent vs. 48 percent), as well as 
placing them in materials that supplement the core reading program during that time period. 

There were no substantive changes from 2004-05 to 2006-07 in teachers’ reported use of 
supports to meet the needs of struggling readers. For example, in both years, approximately 
three-quarters of teachers in RF and non-RF Title I schools reported that their struggling readers 
worked with a more advanced peer within the previous month. 

Most K-3 teachers in both RF and non-RF Title I schools (88-95 percent) reported providing 
additional practice in the previous month to struggling readers in phonemic awareness, phonics, 
and fluency. 

Supports as Reported by Principals and Reading Coaches 

In addition to intervention services, principals also reported on other methods teachers use to 
meet the needs of struggling readers (Exhibit 4.4). In 2006-07, principals and reading coaches in 
both types of schools reported that classroom teachers provided additional practice opportunities 
(98-99 percent) and direct instruction to struggling readers (97 percent) and have trained aides 
working with students during class to provide support (88-90 percent). There were no 
substantive changes from 2004-05 to 2006-07 reported by RF or non-RF Title I schools on these 
activities. 

There were, however, some differences in 2006-07 between RF and non-RF Title I schools in 
the reported staff activities used to meet the needs of struggling readers. Non-RF Title I schools 
reportedly were more likely to use untrained aides or volunteers to work with struggling readers 
than were RF schools (either during the day: 40 percent vs. 23 percent; or after school (18 
percent vs. 10 percent). 
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Exhibit 4.3 



Teachers’ Use of Supports in the Previous Month for Struggiing Readers in RF and Non-RF Titie i 
Schoois, 2004-05 and 2006-07 Schooi Years 



2004-05 



2006-07 



Difference in Difference 




Diagnostic assessment to determine 
core deficits 
-3.9 

Placement in materials that 
supplement the core reading 
program 
- 1.6 

Work with more advanced peer 
0.7 

Special materials for parents 
to provide practice 

4.2 

Work with reading specialist in 
small group 
2.8 

Placement in different level of 
core reading program 
-3.5 

Work with tutor on 
one-to-one basis 
0.5 

Work with reading specialist 
on one-to-one basis 

3.3 

Placement in separate core 
reading program 

1.4 



□ RF Schools 

□ Non-RF Title I Schools 
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75 

73 



64 

63 



59* 
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56 
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57* 



_ 32* 
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Exhibit reads: In 2004-05, 82 percent of teachers in RF schools reported using diagnostic assessment to determine core deficits of 
struggling readers, compared to 61 percent of teachers in non-RF Title I schools. This difference is statistically significant (p < 
.05). In 2006-07, 84 percent of teachers in RF schools reported using this activity, compared to 67 percent of teachers in non-RF 
Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by -4.0 percentage 
points (center column) reflecting that there was a 17 percentage point difference in 2006-07 and a 21 percentage point difference 
in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF teachers and teachers 
in non-RF Title I schools. 

See Appendix B, Exhibit B.4.3 for additional statistics. 

Source: Teacher Survey (Question C9 in 2004-05 and B8 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,684 RF schools and 14,656 non-RF Title I schools. 

Non-response rate across items, groups, and years 0.0 to 0.2 percent. 
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Exhibit 4.4 



Staff Activities to Meet the Needs of Struggiing Readers, as Reported by the Principais or Reading 
Coaches in RF and Non-RF Titie I Schoois, 2004-05 and 2006-07 Schooi Years 
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Classroom teacher provides 
additional practice 
opportunities 
-0.4 

Classroom teacher provides 
additional direct instruction 
0.6 

Trained aides or volunteers 
work with students during class 
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A certified reading specialist 
works directly with students 
-3.6 

Trained aides or volunteers 
work with students before or 
after school 
0.1 

Untrained aides or volunteers 
work with students during class 
-6.7 

Untrained aides or volunteers 
work with students before or 
after school 
-4.8 
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□ RF Schools 
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Exhibit reads: In 2004-05 and 2006-07, 99 percent of RF schools reported that classroom teachers provide additional practice 
opportunities to meet the needs of struggling readers, compared to 98 percent of non-RF Title I schools over the same time period. 
These differences are not statistically significant. The 2004-05 and 2006-07 results differ by 0.4 percentage points (right-hand 
column) reflecting that there was a 1 percentage point difference in 2006-07 and a 1 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.4.4 for additional statistics. 

Source: Reading Coach Survey and Principal Survey (Question El in 2004-05 and 2006-07). 

Weight: School. 

Weighted respondents: Principals or reading coaches in 1,682 RF schools and 14,334 non-RF Title I schools. 

Non-response rate across items, groups, and years: 0 to 2.6 percent. 



Supports for Special Education and ELL Students 

To be successful, reading intervention efforts with struggling readers or ELL students must allow 
time for teachers, ELL staff, or special education teachers to coordinate their instructional 
activities. Absent such coordination — if teachers and other staff are not working on the same 
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reading subskills and in the same sequence — the usefulness of the reading intervention is likely 
to be compromised (National Literacy Panel on Language Minority Children and Youth, 2006). 

Approximately 8 percent of K-3 students in both RF and non-RF Title 1 schools reportedly 
receive special education services. In 2006-07, more than 70 percent of teachers in RF and non- 
RF Title I schools reported that time was set aside to coordinate their reading activities with 
special education staff (Exhibit 4.5). For both groups of teachers, this represents an increase 
from the earlier data collection (for RF teachers from 63 percent to 74 percent and for teachers in 
non-RF Title I schools from 65 percent to 70 percent). 



Exhibit 4.5 

Percent of Teachers Setting Aside Some Time to Coordinate Interventions with Speciai 
Education and Engiish Language Learner Staff in RF and Non-RF Titie i Schoois, 2004-05 and 
2006-07 Schooi Years 

Special Education ELL 




RF Schools Non-RF Title I Schools 



Exhibit reads: In 2004-05, 63 percent of teachers in RE schools reported having some time set aside to coordinate interventions 
for struggling readers with special education staff, compared to 74 percent in 2006-07. This difference is statistically 
significant (p < .05). In 2004-05, 65 percent of teachers in non-RF Title I schools reported having some time set aside, 
compared to 70 percent in 2006-07. This difference is statistically significant (p < .05). 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and 
non-RF Title I schools. 

See Appendix B, Exhibit B.4.5 for additional statistics. 

Source: Teacher Survey (Question C3 in 2004-05 and B3 in 2006-07). 

Weight: School. 

Weighted Respondents: Teachers in 1,096 RF and 8,685 non-RF Title I schools 
Non-response rate across items, groups, and years: < 1 percent. 
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More than 70 percent of RF and non-RF Title I schools have at least one ELL student. However, 
in RF schools the proportion of such students in grades K-3 is higher than in non-RF Title I 
schools (30 percent vs. 20 percent). In 2006-07, in schools serving ELL students, there were no 
significant differences in RF and non-RF Title I schools in the percent of teachers reporting 
coordinating with ELL staff (70 percent). However, this is a significant increase from the 2004- 
OS findings from the 45 percent of RF teachers and 40 percent of teachers in non-RF Title I 
schools who reported such coordination. 

About three-quarters of teachers in RF and non-RF Title I schools reported that their ELL 
students received ESL instruction in the previous month (75 percent vs. 77 percent) and about 
half of these teachers indicated that their ELL students received in-classroom reading help from 
an ELL teacher in the previous month (54 percent vs. 53 percent) (Exhibit 4.6). 



Exhibit 4.6 

Teachers’ Use of Supports in the Previous Month for Engiish Language Learners to Meet the 
Needs of Struggiing Readers, in RF and Non-RF Titie i Schoois, 2004-05 and 2006-07 Schooi 
Years 



2004-05 2006-07 




Exhibit reads: In 2004-05, 70 percent of teachers in RE schools reported using ESL instruction to meet the needs of struggling 
ELL students, compared to 69 percent of teachers in Title I schools. This difference is not statistically significant. In 2006-07, 
75 percent of teachers in RF schools reported using this strategy, compared to 77 percent of teachers in non-RF Title I schools. 
This difference is not statistically significant. The 2004-05 and 2006-07 results differ by -3.3 percentage points (center 
column) reflecting that there was a 2 percentage point difference in 2006-07 and a 1 percentage point difference in 2004-05. 
Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and 
non-RF Title I schools. 

See Appendix B, Exhibit B.4.6 for additional statistics. 

Source: Teacher Survey (Question CIO in 2004-05 and B9 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 971 RF schools and 7,501 non-RF Title I schools. 

Non-response rate across items, groups, and years: < 1 percent. 
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For both RF and non-RF Title I schools with ELL students, there were no substantive changes 
from 2004-05 to 2006-07 in teachers’ reported use of supports to meet the needs of struggling 
readers who are ELLs. For example, in both years, approximately one-fourth of teachers in RF 
and non-RF Title I schools reported that reading instruction was provided in students’ home 
language in the previous month (29 percent vs. 25 percent). 

Non-RF Title I schools were somewhat more likely to rely on certified special education teachers 
to provide recommendations on accommodations for struggling readers than were RF schools 
(84 percent vs. 77 percent). Additionally, non-RF Title I schools were also more likely to use 
bilingual or ESL teachers’ recommendations than were RF schools (52 percent vs. 43 percent). 

Summary 

A central feature of Reading First is to identify, support, and coordinate services for K-3 
students who are at risk of falling behind their peers in their development of reading skills. Staff 
in most RF and non-RF Title I schools reported using a variety of tools to identify struggling 
readers, including reading test scores, teacher or staff recommendations, and classroom 
observations. However, RF schools were reportedly much more likely to use reading coach 
reco mm endations than non-RF Title I schools to identify students for reading interventions, and 
non-RF Title I schools were more likely to use requests from parents than RF schools. 

According to principal reports, intervention services for struggling readers are available in most 
RF and non-RF Title I schools, with an average wait time of three to four school days. Further, 
virtually all RF and Title I schools reportedly provide struggling readers with such supports as 
additional practice opportunities, additional direct instruction, and help from trained aides or 
volunteers. However, a significantly greater percentage of teachers in RF schools reported using 
diagnostic assessments to determine struggling readers’ core deficits, placing students in 
materials that supplement the core reading program, and placing students in intervention 
programs than do teachers in non-RF Title I schools. 

In both RF and non-RF Title I schools, across most activities aimed at helping struggling readers 
there was little change from the 2004-05 to 2006-07 data collections. One exception, however, 
was teachers’ reported coordination of reading interventions with other school staff. From 2004- 
05 to 2006-07, the percent of teachers reporting that time is set aside to coordinate reading 
interventions with ELL staff or special education staff increased dramatically in both RF and 
non-RF Title I schools. 

The survey information provided by non-RF Title I school staff suggests that their activities to 
assist struggling readers are becoming more similar to their RF counterparts. Additionally, 
responses from RF school staff across both data collection periods indicate continued efforts to 
meet the needs of struggling readers. 
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Chapter 5: Assessment 



Assessment of students’ reading proficiency is a central element of Reading First; the legislation 
specifically requires that schools assess students for screening and diagnostic, instructional, and 
outcome purposes (PL 107-110, Title I, Part B, Subpart 1). States and districts are to provide 
assistance to RF schools in selecting, administering, and interpreting the results of reading 
assessments; in addition, states and districts are to provide professional development to teachers 
in the use of reading assessments, particularly with students at risk of reading failure. The 
Reading First program does not advocate the use of any specific assessment but rather requires 
that reading assessments selected by states, districts, or schools be psychometrically strong and 
aligned with instruction (U.S. Department of Education, OESE, Guidance for the Reading First 
Program, April 2002). 

Over the past 15 years, the prominence of assessment and accountability in K-12 education has 
increased greatly, not only for RF schools (Frye, 1999; Goertz & Duffy, 2003). In this section 
we explore the uses of assessment in Reading First schools and non-RF Title I schools during the 
2004—05 and 2006-07 school years. In particular, we describe the selection and interpretation of 
reading assessments, and teachers’ classroom use of reading assessment results. This chapter 
also describes differences between RF and non-RF Title I schools in the types of reading 
assessments teachers reported using most often during the 2006-07 school year and their 
associated administration procedures. 

Key Findings 

While assessment plays an important role in reading programs of both RF and non-RF 
Title I schools, there is evidence that RF schools emphasize assessment somewhat more 
than do non-RF Title I schools. 

In Reading First schools, principals were more likely to report that the state shared 
responsibility for selecting assessments (52 percent vs. 26 percent) and interpreting their 
results (31 percent vs. 12 percent) than were principals in non-RF Title I schools. Further, 
in those sehools that have reading coaches, RF prineipals were also more likely to report that the 
reading eoach shared responsibility for selecting assessments (94 pereent vs. 59 pereent) and 
interpreting their results (51 percent vs. 35 pereent). 

There were significant differences in the types of assessments that teachers in RF schools 
reported using most often compared to teachers in non-RF Title I schools. RF teachers were 
significantly more likely to identify standardized tests as the test they used more often than were 
teachers in non-RF Title I schools. Although rarely reported overall, teachers in non-RF Title I 
schools were significantly more likely to identify informal assessments or report that they do not 
use any assessment than teachers in RF schools. In addition, there were significant differences 
between teachers in RF and non-RF Title I schools’ reports of the specific assessments they use 
most often. Significantly more teachers in RF schools reported using the Dynamic Indicators of 
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Basic Early Literacy Skills (DIBELS) than did teachers in non-RF Title I schools for grouping 
students (43 percent vs. 15 percent), determining student mastery of skills (19 percent vs. 6 
percent), and for identifying core deficits of struggling students (31 percent vs. 14 percent). 

Teachers in RF schools reported administering the assessment they most commonly use 
more often than teachers in non-RF Title I schools and reported that these assessments are 
shorter in length. For example, teachers in RF schools were more likely to report giving the 
test they used for placing or grouping students three or more times during the year than teachers 
in non-RF Title I schools (91 percent vs. 75 percent). Across assessment purposes, significantly 
more RF teachers reported that the assessment they use most often takes 1 5 minutes or less to 
administer than teachers in non-RF Title I schools (e.g., 62 percent vs. 42 percent for placing or 
grouping students). 

RF teachers were more likely to report that using a range of assessment results is central to 
their reading instruction. For example, RF teachers were more likely to report that using test 
results for organizing instructional groups was central to their teaching than teachers in non-RF 
Title I schools (91 percent vs. 78 percent). 

Selection and Interpretation of Reading Assessments 

An effective reading program is expected to have a strong plan for assessing students regularly 
and teachers are expected to use the results of those assessments to inform their reading 
instruction. Toward this end, schools needed to select reading assessments to use for a variety of 
purposes (grouping students, determining student mastery, and identifying core reading deficits) 
in their classrooms. The following section summarizes findings about the receipt of external 
assistance selecting and interpreting assessments, responsibility for selecting and interpreting 
assessments, and the adoption of new reading assessments. 

Receipt of External Assistance 

About half the principals in both RF and non-RF Title I schools received assistance in selecting 
reading assessments (54 percent) (Exhibit 5.1). The proportion of principals in RF schools 
reporting receiving this assistance decreased from 79 percent in 2004-05 to 54 percent in 2006- 
07, while there was no change in the responses of principals in non-RF Title I schools. However, 
RF principals were significantly more likely to report receiving external assistance interpreting 
assessment results than were principals in non-RF Title I schools (75 percent vs. 65 percent)."^^ 
Fewer principals in RF schools indicated receiving this sort of assistance in 2006-07 than in 
2004—05 (75 percent vs. 86 percent). 
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“External” is defined as assistance from persons outside of the school, such as from the district or state, from 
publishers or from university experts. 
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Exhibit 5.1 

Percentage of Principals in RF and Non-RF Title I Schools Reporting Receiving External 




100 80 60 40 20 0 0 20 40 60 80 100 



Percent of Schools Percent of Schools 

Exhibit reads: In 2004-05, 79 percent of RE principals reported that their schools received assistance selecting K-3 reading 
assessments, compared to 56 percent of principals in non-RF Title I schools. This difference is statistically significant (p < 
.05). In 2006-07, 54 percent of principals in both RF and Title I schools reported receiving this type of assistance. The 2004- 
05 and 2006-07 results differ by -23.3 percentage points (right-hand column) reflecting the fact that there was a 0 percentage 
point difference between RF and non-RF Title I schools in 2006-07 and a 23.3 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.5.1 for additional statistics. 

Source: Principal Survey (Question B7 in 2004-05 and B2 in 2006-07). 

Weight: Principal. 

Weighted respondents: Principals in 1,684 RF schools and 14,652 non-RF Title I schools. 

Non-response rate: < 1 percent. 



Responsibility for Selection of Reading Assessments and Interpretation of the Results 

Significantly more principals in RF than non-RF Title I schools reported that the state had some 
responsibility for seleeting assessment instruments (52 pereent vs. 26 pereent) (Exhibit 5.2). The 
larger role of the state in RF schools could be due to the role of the state in the Reading First 
applieation proeess, during whieh states needed to deseribe their plan for assessing students’ 
progress. In these plans, states may have identified specifie assessments that RF schools in their 
states would use. In 2006-07, three-quarters of principals in both RF and non-RF Title I sehools 
reported that the district has some responsibility for selecting assessments. 

Nearly all principals in RF schools with reading eoaches indicated that reading coaehes share 
responsibility for interpreting assessment results, whereas significantly fewer principals in non- 
RF Title I schools that have reading eoaehes reported that their coaehes had that responsibility 
(98 pereent vs. 77 percent). Further, nearly all principals (94 percent) in both types of schools 
identified themselves as responsible for interpreting assessment results. 
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Exhibit 5.2 



Responsibiiity for Reading Assessment Activities in RF and Non-RF Titie I Schoois as Reported by 
Principais, 2004-05 and 2006-07 Schooi Years 
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Exhibit reads: In 2004-05, 27 percent of RE principals reported that the state was, in part, responsible for the selection of reading 
assessments, compared to 13 percent of principals in non-RF Title I schools. This difference is statistically significant (p < .05). In 
2006-07, 3 1 percent of RF principals reported that the state was responsible for this activity, compared to 12 percent of principals in 
non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results are not significantly 
different, as indicated by the non-significant 4.9 percentage point difference-in-difference. 

Notes: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. The reading coach results include only those schools that have coaches. 

See Appendix B, Exhibit B.5.2 for additional statistics. 

Source: Principal Survey (Question D4 in 2004-05 and D2 in 2006-07). 

Weight: Principal. 

Weighted respondents: Principals in 1,684 RF schools and 14,656 non-RF Title I schools. 

Non-response rates across survey items: < 1 percent. 
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Adoption of New Assessments 



Only a small proportion of RF schools reported adopting new reading assessments in their third 
year of implementation (2006-07), significantly fewer than in non-RF Title I schools (12 percent 
vs. 33 pereent). This is not surprising, given that the majority of these RF sehools (77 pereent) 
reported adopting new reading assessments in 2004-05, when they were in their first year of 
implementation. Only 9 pereent of RF sehools reported adopting new assessments in both 2004- 
OS and 2006-07. 

Dedicated Time for Teachers to Use Assessment Data to Pian instruction 

Regular assessment of students’ reading skills allows teaehers to modify instruction to better 
meet the needs of their students. Presumably, teachers are more able to meet this goal if they 
have time regularly set aside to review the results of assessments and plan their instruction 
accordingly."^^ Significantly more teachers in RF sehools reported that time was set aside 
regularly to use assessment data to plan instruetion than teaehers in non RF Title I sehools (89 
pereent vs. 79 pereent) (Exhibit 5.3). 



Exhibit 5.3 



Time Set Aside for K-3 Teachers to Use Assessment Data to Pian Instruction in RF and Non-RF 
Titie I Schoois, 2004-05 and 2006-07 Schooi Years 
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Exhibit reads: In 2004-05, 28 percent of RE teachers reported that time was set aside once a week or more to use assessment data 
to plan instruction, compared to 23 percent of teachers in non-RF Title I schools. This difference is statistically significant (p < 
.05). In 2006-07, 33 percent of RF teachers reported that such time was set aside compared to 27 percent of teachers in non-RF 
Title I schools; this difference is also statistically significant (p < .05). 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non- 
RF Title I schools. 

See Appendix B, Exhibit B.5.3 for additional statistics. 

Source: Teacher Survey (Question C3 in 2004-05 and B3 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,684 RF schools and 14,656 non-RF Title I schools. 

Non-response rate: < 1 percent. 



For the purposes of this report, we defined time on a regular basis to mean at least one to four times per year 
and frequently as once a week or more. 
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Reading Assessments Used by Teachers 



This section compares responses of teachers in RF and non-RF Title I schools with regard to 
their use of assessments as part of their reading programs. For each purpose listed, teachers 
identified the one assessment they used most often.^' In addition, the section describes variation 
between RF and non-RF Title I schools in the frequency of assessments, their duration, format, 
and use for accountability purposes. 

Types of Assessments 

Across assessment purposes, teachers in RF schools were more likely to report that they used a 
standardized assessment most often than were teachers in non-RF Title I schools (Exhibit 5.4). 
For example, a significantly higher percentage of RF teachers reported using a standardized 
assessment most often to place or group students than were teachers in non-RF Title I schools 
(64 percent vs. 45 percent). Also, across assessment purposes, very few teachers in either RF or 
non-RF Title I schools reported using district-specific, state-specific, or informal assessments as 
the test used most often. 

Teachers in both RF and non-RF Title I schools reported using assessments from the core, 
supplementary, or intervention reading program to determine student mastery of skills (3 1 
percent and 30 percent) more so than for placing or grouping students (11 percent and 15 
percent) or for identifying the core deficits of struggling students (14 percent and 1 1 percent, 
respectively). 

The assessments teachers identified using most often were generally consistent across grade 
levels. We did find, however, that kindergarten teachers in both RF and non-RF Title I schools 
were more likely to report using informal assessments (structured or unstructured) for 
determining student mastery than were teachers in grades 1-3 (6 and 23 percent, respectively, for 
RF and non-RF Title I kindergarten teachers, and 2-4 percent and 7-12 percent, respectively, for 
RF and non-RF Title I grade 13 teachers). In addition, the percentage of teachers in RF and 
non-RF Title I schools who reported that they most often used assessments from the reading 
program for determining student mastery increased with grade progression from kindergarten 
(18 percent for RF teachers and 17 percent for teachers in non-RF Title I schools) to second 
grade (32 percent for RF teachers and 30 percent for teachers in non-RF Title I schools). 



This question changed in structure and content from 2005 to 2007. Therefore, these data cannot be compared 
across survey years. We revised the question in order to increase our ability to categorize teachers’ open-ended 
responses (approximately 40 percent of responses were not appropriate for analysis in 2005), as well as to 
decrease the non-response rate (approximately 10 percent in 2005). 

On the 2006-07 survey, although we asked teachers to list the one assessment they used most often, some 
teachers listed more than one assessment per purpose. We excluded these cases from the analysis of that 
purpose. Approximately 20 percent of teachers’ responses were excluded from analysis. 

See Appendix B, Exhibits B.5.4a-d for additional grade level statistics. 
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Exhibit 5.4 



Types of Assessments Used Most Often by Teachers in RF and Non-RF Title 1 Schools, by 
Assessment Purpose, 2006-07 School Year 


Assessment Purpose 




Teachers ^ ^ 


Type of Assessment 


RF 


Non-RF Title 1 


Placing or grouping of students 


Formal assessments 






Core, supplementary, or intervention reading 






program assessment 


11% 


15%* 


District-specific assessment 


2 


2 


State-specific assessment 


2 


3 


Other standardized assessment 


64* 


45 


Informal assessments 






Structured informal assessments 


3 


7* 


Unstructured informal assessments 


2 


5* 


Assessment is not used for this purpose 


2 


7* 


Determining student mastery of skills 


Formal assessments 






Core, supplementary, or intervention reading 






program assessment 


31 


30 


District-specific assessment 


5 


4 


State-specific assessment 


2 


4 


Other standardized assessment 


35* 


22 


Informal assessments 






Structured informal assessments 


1 


5* 


Unstructured informal assessments 


4 


10* 


Assessment is not used for this purpose 


1 


5* 


Identifying the core deficits of struggling students 


Formal assessments 






Core, supplementary, or intervention reading 






program assessment 


14 


11 


District-specific assessment 


2 


2 


State-specific assessment 


2 


3 


Other standardized assessment 


51* 


34 


Informal assessments 






Structured informal assessments 


3 


8* 


Unstructured informal assessments 


4 


6* 


Assessment is not used for this purpose 


6 


18* 



Exhibit reads: In 2006-07, 1 1 percent of RE teachers reported that they used assessments from the core or supplementary reading 
program most often for placing or grouping students, compared to 15 percent of teachers in non-RF Title I schools. This difference is 
statistically significant (p < .05). 

^ Percentages by grade or assessment purpose will not add up to 100 percent due to non-response, responses that we were not able to 
categorize, and multiple responses. 

^ We were not able to categorize about 8 percent of teachers’ responses. “Not able to categorize” includes responses that were too 
vague to be coded (e.g., assessment, test, eight-weeks, benchmarks, phonics, rubric, pretest, quarterly test, fluency) and responses 
that were not discemable (e.g., ELLA stands for Early Learning Literacy in Arkansas or the English Language and Literacy 
Assessment). In addition, 10 percent of teachers’ responses were excluded from analyses because the teacher named two or more 
assessments that we could categorize; however, we could not identify the one assessment used most often. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RE and non-RE 
Title I schools. 

See Appendix B, Exhibits B.5.4a-d for additional grade level statistics. 

Source: Teacher Survey (Questions B6 in 2006-07). 

Weight: Teacher. 

Weighted respondents: 22,854 teachers in RE schools and 177,157 teachers in non-RE Title I schools. 

Nonresponse rates across survey items: 2.3 to 4.7 percent. 
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Overall, few teachers reported that they do not use an assessment for the purposes listed. 
However, given that the Reading First program specifically requires testing in reading in the 
early grades, it is not surprising that teachers in RF schools report this significantly less often 
than teachers in non-RF Title I schools across assessment purposes. The largest number of 
teachers across grades reported that they did not use an assessment for identifying the core 
deficits of struggling students (6 percent in RF schools vs. 18 percent in non-RF Title I schools). 
These findings are generally consistent across grades. 

Specific Assessments 

For each assessment purpose, teachers named a myriad of specific formal assessments. The most 
commonly named assessments were:^^ 

• Dynamic Indicators of Basic Early Literacy Skills (DIB ELS), 

• Developmental Reading Assessment (DRA), and 

• Texas Primary Reading Inventory (TPRI). 

Across all assessment purposes, significantly more teachers in RF schools reported that they used 
the DIBELS most frequently for grouping students (43 percent vs. 15 percent), determining 
student mastery of skills (19 percent vs. 6 percent), and identifying core deficits of struggling 
students (31 percent vs. 14 percent) than did teachers in non-RF Title I schools (Exhibit 5.5). 

The specific assessments teachers identified using most often were also generally consistent 
across grade levels. However, the percentage of teachers in RF and non-RF Title I schools who 
reported that the DIBELS was the test they used most often generally decreased with grade 
progression. As an example, for determining student mastery of skills, 30 percent of RF 
kindergarten teachers reported the DIBELS as the assessment they use most often, which fell to 
23 percent in first grade, to 10 percent in second grade, and to 8 percent in third grade. This 
same pattern generally holds across assessment purposes in both RF and non-RF Title I 
schools. 



No informal assessment was identified by at least 5 percent of all RF and teachers in non-RF Title I schools. 
See Appendix B, Exhibits B.5.5a-d for additional grade level statistics. 
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Exhibit 5.5 



Types of Specific Assessments Reported as Used Most Often by K-3 Teachers in RF and Non-RF 
Title I Schools, by Assessment Purpose, 2006-07 School Year 



Assessment Purpose 




Teachers ^ ^ 


Type of Assessment 


RF 


Non-RF Title 1 


Placing or grouping of students 


Formal assessments 






DIBELS 


43%* 


15% 


DRA 


4 


13* 


TPRI 




4 


Determining student mastery of skills 


Formal assessments 






DIBELS 


19* 


6 


DRA 


1 


4* 


TPRI 


6* 


2 


Identifying the core deficits of struggling students 


Formal assessments 






DIBELS 


31* 


14 


DRA 


1 


4* 


TPRI 


9* 


4 



Exhibit reads: In 2006-07, 43 percent of RE teachers identified the DIBELS as the reading assessment they use most often for 
placing or grouping students, compared to 15 percent of teachers in non-RF Title I schools. This difference is statistically 
significant (p < .05). 

^ Only individual assessments that constituted more than 5 percent of responses are included in this exhibit.^ Percentages by grade 
or assessment purpose will not add up to 100 percent due to non-response, responses that we were not able to categorize, and 
multiple responses. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non- 
RF Title I schools. 

See Appendix B, Exhibits B.5.5a-d for additional grade level statistics. 

Source: Teacher Survey (Question B6 in 2006-07). 

Weight: Teacher. 

Weighted respondents: 23,921 teachers in RF schools and 181,283 teachers in non-RF Title I schools. 

Nonresponse rates across survey items: < 1 percent. 



Administration of Assessments 

Number of Assessment Administrations 

Frequency of assessment administration varied across RF and non-RF Title I schools as well as 
by assessment purpose (Exhibit 5.6). Across all three assessment purposes, RF teachers were 
significantly more likely than were teachers in non-Title I schools to report administering their 
most useful assessment five or more times per year for placing students into groups (38 percent 
vs. 21 percent), determining student mastery (60 percent vs. 51 percent), and identifying the core 
deficits of struggling students (53 percent vs. 39 percent). 

In both RF and non-RF Title I schools, teachers reported assessing students most often to 
determine mastery of skills; the majority of teachers in both types of schools reported assessing 
students five or more times for this purpose (60 percent and 51 percent). Alternatively, teachers 
reported testing somewhat less often to place or group students; about half of teachers in both RF 
and non-RF Title I schools reported testing three to four times per year for this purpose. 
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Exhibit 5.6 



Number of Times Most Frequently Used Assessment is Administered Per Year as Reported by 
Teachers in RF and Non-RF Title I Schools, by Assessment Purpose, 2006-07 School Year 



Percent of 
Teachers 




RF Non-RF 
Title I 



RF Non-RF 
Title I 



RF Non-RF 
Title I 



Placing or grouping Determining student Identifying the core 

students mastery of skills deficits of struggling 

readers 

Exhibit reads: In 2006-07, 38 percent of RE teachers reported that they administer the assessment they use most often for placing or 
grouping students five or more times per year, compared to 21 percent of the teachers in non-RF Title schools. This difference is 
statistically significant (p < .05). 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.5.6, for additional statistics. 

Source: Teacher Survey (Question B6 in 2006-07). 

Weight: Teacher. 

Weighted respondents: 22,350 teachers in RF schools and 163,268 teachers in non-RF Title I schools. 

Non-response rates across survey items: 1.9 to 7.3 percent. 



Time Required for Assessments 

While teachers in RF schools reported assessing students more frequently, overall, Reading First 
teachers reported administering assessments that require less time than the assessments teachers 
administered in non-RF Title I schools (Exhibit 5.7). Across assessments purposes, RF teachers 
were significantly more likely to report that the assessment they use most often takes 15 minutes 
or less to administer than were teachers in non-RF Title 1 schools. For example, 63 percent of 
RF teachers reported using an assessment to group students that takes 1 5 minutes or less 
compared to 42 percent of teachers in non-RF Title I schools. 
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Exhibit 5.7 



Length of Administration for Most Frequentiy Used Assessment as Reported by Teachers in RF and 
Non-RF Title I Schools, by Assessment Purpose, 2006-07 School Year 



Percent of 
Teachers 




RF Non-RF 
Title I 



RF Non-RF 
Title I 



RF Non-RF 
Title I 



Placing or grouping Determining student Identifying the core 

students mastery of skills deficits of struggling 

readers 

Exhibit reads: In 2006-07, 17 percent of RF teachers reported that they most often use an assessment for placing or grouping 
students that takes more than 30 minutes to administer, compared to 27 percent of the teachers in non-RF Title I schools. This 
difference is statistically significant (p < .05). 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.5.7 for additional statistics. 

Source: Teacher Survey (Questions B6 in 2006-07). 

Weight: Teacher. 

Weighted respondents: 21,941 teachers in RF schools and 160,830 teachers in non-RF Title I schools. 

Non-response rates across survey items: 3.3 to 14.8 percent. 



There was also variation across assessment purposes in the amount of time required for 
administering assessments, with student mastery tests taking the longest (Exhibit 5.7). About 40 
percent of in both RF and non-RF Title I schools were likely to report using assessments 
requiring more than 30 minutes for this purpose. 

Perhaps not surprisingly, given developmental differences in attention span, the length of 
assessments that teachers administer in RF and non-RF Title I schools increases with grade 
progression. For example, to assess student mastery, 60 percent of RF and 47 percent of non-RF 
Title I kindergarten teachers administer tests that require 15 minutes or less, while at third grade, 
to assess this skill only 24 percent third-grade teachers in RF schools and 16 percent in non-RF 
Title I schools reported using such short assessments. 
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Accountability Purposes for Assessments 

In addition to providing teachers information for planning and modifying instruction, 
assessments also provide information relevant for school accountability requirements. Not 
surprisingly, there are significant differences between teachers in RF and non-RF Title I schools 
in their reports of whether the assessment they reported administering most often is used for 
accountability purposes for the Reading First program. No Child Left Behind {NCLB), or another 
program (Exhibit 5.8). The vast majority of RF teachers (86 percent) reported that across the 
three assessment purposes, the assessment they administered most often was used for 
accountability purposes for the Reading First program. Eighteen percent of the teachers in non- 
RF Title I schools also reported that the purpose of the test was for accountability for Reading 
First; it is not clear why teachers in non-RF Title I schools would provide this response. 

In contrast, about two-thirds of teachers in non-RF Title I schools reported that their most 
frequently administered assessment is used to meet other programs’ accountability requirements 
(65 percent) and nearly half indicated that the assessment results were used for NCLB 
accountability purposes (45 percent). 



Exhibit 5.8 

Accountability Purpose for Most Frequently Used Assessment as Reported by Teachers in RF 
and Non-RF Title I Schools, 2006-07 School Year 

Teachers 



Accountability Purpose 


RF 


Non-RF Title I 


Reading First 


96%* 


19% 


NCLB 


37 


47* 


Other Program 


41 


63* 



Exhibit reads: In 2006-07, 86 percent of RF teachers reported administering the assessment they use most often for Reading 
First accountability purposes, compared to 18 percent of teachers in non-RF Title I schools. This difference is statistically 
significant (p < .05). 

Notes: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and 
non-RF Title I schools. Column percent totals exceed 100 percent because respondents could indicate that an assessment was 
used for multiple accountability purposes. 

See Appendix B, Exhibit B.5.8 for additional statistics. 

Source: Teacher Survey (Question B6 in 2006-07). 

Weight: Teacher. 

Weighted respondents: 23,503 teachers in RF schools and 176,247 teachers in non-RF Title I schools. 

Non-response rates across survey items: 1.5 to 1.9 percent. 



Teachers’ Classroom Use of Reading Assessments Results 

In 2006-07, teachers in RF schools were more likely to rate the use assessment results as central 
to their instruction across a variety of purposes than were teachers in non-RF Title I schools 
(Exhibit 5.9). This is especially the case for several uses of assessment results explicitly 
described in the Reading First guidance (in bold in Exhibit 5.9); to organize instructional groups 
(91 percent vs. 78 percent), to determine progress on skills (88 percent vs. 80 percent), and to 
identify students who are struggling and need intervention services (83 percent vs. 69 percent). 
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Exhibit 5.9 

Teachers Use of Assessments in RF Schoois and Non-RF Titie i Schoois, 2004-05 and 2006-07 
Schooi Years 



2004-05 



"86 



71 



*85 



77 



*81 



64 

61 

61 



*58 



47 



*56 



44 



53 

52 



100 



— I — 

80 



— I — 

60 



— I — 

40 



— I — 

20 



Percent of Teachers 



Difference in Difference 



Use of Assessments 

Use the test results to organize 
instructional groups 

- 2.2 



Use tests to determine 
progress on skills 

0.1 

Use diagnostic tests to identify 
students who need reading 
intervention services 

- 3.0 

Use informal reading inventories 
1.8 

Use screening test to identify 
students who need a 
supplementary reading program 
1.0 

Use test to determine who can 
benefit from the core reading 
series 

3.5 

Conduct miscue analysis, 
analyzing errors students make 
while reading aloud 

3.6 







78 










80 








83 



2006-07 



□ RF Schoois 

□ Non-RF Titie I Schoois 



91* 



62 

60 

61* 



50 



59* 



43 



56* 



51 



88 * 



69 



20 40 60 80 

Percent of Teachers 



100 



Exhibit reads: In 2004-05, 86 percent of RF teachers reported that they used test results to organize instructional groups, compared 
to 71 percent of teachers in non-RF Title I schools. This difference is statistically significant (p < .05). In 2006-07, 91 percent of 
RF teachers reported that they use test results for this purpose, compared to 78 percent of teachers in non-RF Title I schools. This 
difference is also statistically significant (p < .05). The 2006-07 and 2004-05 results are not different from each other, as indicated 
by the small, non-significant difference-in-difference of 2.2 percentage points. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.5.9 for additional statistics. 

Source: Teacher Survey (Question C6 in 2004-05 and B5 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,684 RF schools and 14,656 non-RF Title I schools. 

Non-response rates across survey items: < 1 percent. 
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Also, RF teachers were more likely to rate several assessment uses that not mentioned in the 
guidance as central than were teachers in non-RF Title I schools: to determine who can benefit 
from the core reading series (59 percent vs. 41 percent), and to identify students who need a 
supplementary reading program (61 percent vs. 50 percent). In general, teachers’ uses of 
assessment results were similar in 2004-05. 

Summary 

The Reading First program strongly emphasizes the role of assessments in K-3 reading 
programs. There are some key differences between RF and non-RF Title I schools in their use of 
assessments to inform their reading programs in 2006-07. However, assessment plays a 
prominent role in both RF and non-RF Title I schools. 

RF and non-RF Title I schools differ in who has responsibility for choosing assessments, the 
types of assessments teachers report using most often, the frequency and length of the 
assessment used most often, and the extent to which the use of assessment results is central to 
reading instruction. While there are significant differences between the groups of schools, 
overall, both groups report engaging in high levels of assessment related practices that are 
consistent with Reading First. 

In 2006-07, when RF schools were in their third year of implementation, few RF principals 
reported adopting new reading assessments; nonetheless, they were more likely to report that the 
state and reading coach shared responsibility for selecting assessments than principals in non-RF 
Title I schools. RF principals were also more likely to report that the state shared responsibility 
for interpreting assessment results, and that they received external assistance interpreting 
assessment results than principals in non-RF Title I schools. 

The vast majority (over 75 percent) of teachers in both types of schools indicated that regular 
time was set aside to use assessment results to plan instruction. However, significantly more 
teachers in RF schools than non-RF Title I schools reported that this was the case in their school. 
In addition, RF teachers were more likely to report that using assessment results was a central 
element of their reading instruction than were teachers in non-RF Title I schools. Again, this 
difference notwithstanding, the majority of non-RF teachers also report using assessment results 
to inform their reading instruction. 

Teachers in RF and non-RF Title I schools differed significantly in the types of assessments they 
reported using most often to place students, determine student mastery, and identify struggling 
students’ core deficits. RF teachers were significantly more likely than teachers in non-RF 
Title I schools to report using standardized assessments for these purposes, and teachers in non- 
RF Title I schools were significantly more likely to report that they do not use an assessment 
than were RF teachers. 

Three specific assessments were identified most often by teachers from both types of schools: 
the Dynamic Indicators of Basic Early Literacy Skills (DIBELS), Developmental Reading 
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Assessment (DRA), and the Texas Primary Reading Inventory (TPRI). For each assessment 
purpose, RF teachers were significantly more likely to report that they used the DIBELS and the 
TPRI than were teachers in Title I schools. In contrast, across assessment purposes, teachers in 
non-RF Title I schools were significantly more likely to report that they used the DRA than were 
RF teachers. 

Teachers in RF and non-RF Title I schools also differed significantly in the number and length of 
the assessments they use most often. RF teachers reported administering their most useful 
assessments more often than teachers in non-RF Title I schools In addition, RF teachers were 
more likely to report that these assessments take 1 5 minutes or less to administer than teachers in 
non-RF Title I schools. Finally, the virtually all RF teachers (96 percent) reported that the 
assessment they administer most often is used for Reading First program accountability 
purposes. In contrast, teachers in non-RF Title I schools were more likely than teachers in RF 
schools to report that their most frequently administered assessment is used to meet another 
(unspecified) program’s accountability requirements (63 percent vs. 41 percent). 
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Chapter 6: Oversight and Classroom Support 
Activities 



It is teachers who provide reading instruction directly to their students, yet other educators also 
support and supervise teachers’ reading instruction. One model for supporting teachers’ reading 
instruction involves hiring a reading coach to help teachers develop the skills needed to 
implement reading instruction aligned with scientifically based reading instruction. Teachers 
also receive direct classroom support for their instruction either from coaches, their peers or 
reading specialists. In addition to these types of support, oversight of the reading program 
typically is provided by school principals, district staff, and state personnel. 

In this chapter, we present findings on the support teachers receive for their reading programs, 
both in terms of the role reading coaches play in providing this support and in terms of other 
forms of direct classroom support provided to teachers. Second, we present findings about the 
oversight provided at the principal, district, and state levels. 

Key Findings 

Reading programs in RF schools are more likely to receive support from reading coaches 
and oversight from principals and the state than programs in non-RF Title I schools. 

RF schools were significantly more likely to have a reading coach than were non-RF Title I 
schools (99 percent vs. 57 percent), reflecting the fact that nearly all states required RF 
schools to have a reading coach. Non-RF Title I schools have no such requirement at the 
federal level. 

In many states reading coaches are expected to work primarily with teachers in implementing 
their reading programs rather than providing direct instruction to students.^^ Responses from 
reading coaches in RF schools indicate that this expectation is being met. Reading coaches in 
RF schools were significantly less likely than coaches in non-RF Title I schools to report 
that providing direct instruction to students is central to their work (28 percent vs. 49 
percent). Further, reading coaches in RF schools reported spending much more time in the role 
of a reading coach than did coaches in non-RF Title I schools, with 75 percent reporting that they 
spend all their time in this role compared to only 19 percent in non-RF Title I schools. 



The Reading First guidance indicates that reading coaches should be included as part of the required 
professional development strategy that is part of the request for Reading First funding (U.S. Department of 
Education, 2002, page 26). Many states have defined the role of a reading coach as working with and 
supporting teachers in their reading instruction. For example, the Idaho State Department of Education states, 
“The reading coach serves as a resource for teachers to assist them in the implementation of scientifically based 
programs, data analysis, intervention needs, and differentiating instruction for their students” 
(http://www.sde.idaho.gov/readingfirst/, retrieved on 4/16/08). 
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Reading coaches in Reading First schools were significantly more likely than coaches in 
non-RF Title I schools to rate a variety of instructional support activities as central to their 
work. Significantly more reading coaches in RF schools than in non-RF Title 1 sehools rated as 
central to their work: assisting teachers in using the core reading program (89 pereent vs. 60 
pereent), forming instruetional groups (88 pereent vs. 68 pereent), and monitoring the 
effectiveness of strategies for struggling readers (93 percent vs. 77 pereent). However, more 
than half of reading eoaches in non-RF Title I schools rated most aetivities as eentral to their 
work, and over three-quarters rated designing strategies for struggling readers (83 percent) and 
monitoring their effectiveness (77 pereent) as central to their work. 

Teachers in RF schools were more likely to report receiving ongoing, direct support for 
teaching reading than were teachers in non-RF Title I schools, such as: interpretation of 
assessment data (91 pereent vs. 70 percent), assistanee from a reading coaeh or specialist in 
diagnosing individual student needs (72 pereent vs. 48 percent) or intervention service help for 
individual students (73 percent vs. 52 pereent). 

Reading coaches in RF schools were significantly more likely than non-RF Title I coaches 
to rate a variety of administrative support activities as central to their work. A greater 
proportion of reading eoaches in RF sehools than in non-RF Title I sehools rated as eentral to 
their work, aetivities such as compiling reading assessment data (92 pereent vs. 73 percent) and 
ordering or managing reading instruetion materials (75 pereent vs. 61 pereent). Notwithstanding 
these differenees, the majority of reading coaches in non-RF Title I schools rated the following 
activities as eentral to their work: participation in professional development (85 percent); 
administering reading assessments (71 pereent); and eompiling reading assessment data (73 
pereent). 

Principals’ survey responses indicate that the state plays a more significant role in the 
oversight of reading programs in RF schools than in non-RF Title I schools across a variety 
of activities, including monitoring the implementation of the reading program (43 percent vs. 7 
percent), interpretation of assessment results (26 percent vs. 12 percent) and selection of 
professional development topics (20 percent vs. 5 percent). 

Support for Reading Programs 

Reading Coaches 

Reading First schools in nearly all states are required to have a reading coordinator, often called 
a reading coach, who is responsible for helping teachers implement activities aligned with 
SBRR. In 2006-07, virtually all Reading First schools had a designated reading coach (99 



The Reading First guidance specifically indicates that the state education agency “. . .must assess and evaluate, 
on a regular basis, the progress of local educational agencies that receive subgrants in meeting the goals of the 
Reading First program” (page 19). 

74 Chapter 6: Oversight and Classroom Support Activities 





percent), compared to 57 percent of non-RF Title I schools. Over 90 percent of reading 
coaches in both RF and non-RF Title I schools reported working in just one school (Exhibit 6.1). 

Although most coaches in RF and non-RF Title I schools reported spending the majority of their 
time in one school, there are large and consistent differences in the amount of time coaches 
reported spending in the role of reading coach. In 2006-07, three-quarters of coaches in RF 
schools reported spending all of their time in the role of reading coach, compared to only 19 
percent of coaches in non-RF Title I school. 

Responsibilities of Reading Coaches 

Reading coaches rated a series of different activities as central to their roles and 
responsibilities.^^ As noted earlier, in Reading First schools, reading coaches are expected to 
work primarily with teachers in implementing their reading programs rather than providing direct 
instruction to students. Thus, as we might expect, a smaller percentage of reading coaches in RF 
schools reported that providing direct reading instruction to students is central to their work than 
did coaches in non-RF Title I schools (28 percent vs. 49 percent) (Exhibit 6.2). These 
differences between RF and non-RF Title I suggest that the role of “reading coach” in non-RF 
Title I schools may, in some schools, be that of the traditional “reading specialist.” 



Because respondents from non-RF Title I schools who completed the Reading Coach Survey reported a 
multiplicity of job titles, we used their responses to two survey questions to determine their inclusion in the 
comparison group of reading coaches: How central is each of the following activities? 1) “Coaches staff on a 
range of topics”; and 2) “Organizes professional development for K-3 teachers.” Respondents who answered a 
3 (“somewhat central”) or above (on a five-point scale) for at least one of these two items were included in the 
comparison group of reading coaches from non-RF Title I schools for these sets of analyses regarding reading 
coach responsibilities (Exhibits 6.1, 6.2, 6.3, 6.4). As a result, 50 reading coach respondents from non-RF Title 
schools were excluded from these analyses. 

Respondents rated items on a five point scale ranging from 1 (“Do no do or not at all central”), through 3 
(“somewhat central”), to 5 (“absolutely central”). Items rated 4 or 5 were considered “central” in this analysis. 
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Exhibit 6.1 



Responsibilities of Reading Coaches in RF and Non-RF Title I Schools, 2004-05 and 2006-07 
School Years 





2004- 


-05 


2006- 


-07 


2006-07 


Responsibilities 


RF 

Schoois 


Non-RF 
Titie i 
Schoois 


RF 

Schoois 


Non-RF 
Titie i 
Schoois 


Difference- 

2004-05 

Difference 


Average number of schools with which 
reading coach works 


1.2 


1.4* 


1.1 


1.2 


0.2 


... 1 school 


91 %* 


84% 


95% 


91% 


-3.2% 


... 2 schools 


6 


9 


4 


5 


1.1 


... 3 schools 


1 


1 


1 


1 


-0.1 


... 4+ schools 


2 


6* 


0 


2 


2.1 


Average number of teachers with whom 
reading coach works 


19.5 


23.3* 


19.1 


22.1* 


0.8 


... 1-10 teachers 


18% 


16% 


19% 


18% 


-0.9% 


... 11-20 teachers 


45* 


34 


43* 


30 


2.3 


... 21-30 teachers 


24 


29 


25 


32 


-1.5 


... 31+ teachers 


13 


22* 


12 


20* 


0.1 


Average percentage of time spent as 
reading coach in this schooi 


87%* 


59% 


91 %* 


55% 


7.4%* 


... 100% time 


67%* 


19% 


75%* 


19% 


8.2% 


... 75-99% time 


15 


16 


11 


18 


-5.0 


... 50-74% time 


12 


37* 


9 


37* 


-2.9 


... 25-49% time 


5 


17* 


4 


12* 


4.0 


... 1-24% time 


2 


10* 


2 


14* 


-4.3 



Exhibit reads: In 2004-05, reading eoaehes in RF sehools reported working with an average of 1.2 sehools, eompared to 1.4 



sehools for reading eoaehes in non-RF Title I sehools, a statistieally signifieant differenee (p < .05). In 2006-07, reading eoaehes 
in RF sehools reported working with an average of 1.1 sehools, eompared to 1.2 sehools for reading eoaehes in non-RF Title I 
sehools. The 2004-05 and 2006-07 results differ by 0.2 sehools (right-hand eolumn) refleeting the faet that there was a -0.1 
sehool differenee between RF and non-RF Title I sehools in 2006-07 and a -0.3 sehool differenee in 2004-05. 

Note: The stars (*) indieate that there is a statistieally signifieant differenee (i.e., p-value < .05) between RF and non-RF Title I 
sehools. 

See Appendix B, Exhibit B.6.1 for additional statisties. 

Souree: Reading Coaeh Survey (Questions B1-B3 in 2004-05 and 2006-07). 

Weight: Reading eoaeh. 

Weighted respondents: Reading eoaehes in 1,632 RF sehools and 7,303 non-RF Title I sehools. 

Non-response rate aeross items, years, and groups: 0 to 15.3 pereent. 



A greater percentage of reading coaches in RF schools rated several other teacher support 
activities as central to their work, including coaching staff on a range of topics (95 percent vs. 72 
percent), providing training or professional development (94 percent vs. 73 percent), and 
organizing professional development (86 percent vs. 61 percent), than in non-RF Title I schools 
(Exhibit 6.2). It is noteworthy that the percentage of coaches in non-RF Title I schools reporting 
that coaching staff on a variety of topics was central to their work decreased from 83 percent in 
2004—05 to 72 percent in 2006-07. A similar decrease occurred for providing professional 
development in reading materials, from 87 percent in 2004-05 to 73 percent in 2006-07. Over 
the same time period, there was no change on these activities as reported by coaches in RF 
schools; in both time periods more than 90 percent rated these teacher support activities as 
central to their work. 
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Exhibit 6.2 



Percentage of Reading Coaches Rating Various Teacher Support Activities as Centrai to Their Work 
in RF and Non-RF Titie i Schoois, 2004-05 and 2006-07 Schooi Years 



*91 



83 



*88 



*93 



87 



*69 



67 



2004-05 



47 



32 



*53 



100 



— I — 

80 



— I — 

60 



— I — 

40 



— I — 

20 



Percent of Schools 



2006-07 



Difference in Difference 




□ RF Schools 






□ Non-RF Title 1 Schools 



Facilitate grade level 
meetings 
6.3 

Coach staff on a range 
of topics 
15.8* 

Provide direct reading 
instruction to students 
1.1 

Organize professional 
development for K-3 teachers 
4.0 

Provide training/professional 
development in reading 
materials, strategies, and 
assessments 
13.8* 



28 



79* 



51 



95* 



72 



49* 



86 * 



61 



94* 



73 



— I — 

20 



— I — 

40 



— I — 

60 



— I — 

80 



100 



Percent of Schools 



Exhibit reads: In 2004-05, 69 percent of reading coaches in RE schools reported that facilitating grade level meetings was central to 
their work, compared to 47 percent of reading coaches in non-RF Title I schools, a statistically significant difference (p < .05). In 
2006-07, 79 percent of reading coaches in RF schools reported that facilitating grade level meetings was central to their work, 
compared to 51 percent of reading coaches in non-RF Title I schools, a statistically significant difference (p < .05). The 2004-05 
and 2006-07 results differ by 6.3 percentage points (right-hand column) reflecting the fact that there was a 28 percentage point 
difference between RF and non-RF Title I schools in 2006-07 and a 22 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.6.2 for additional statistics. 

Source: Reading Coach Survey (Question B4 in 2005 and 2007). 

Weight: Reading coach. 

Weighted respondents: Reading coaches in 1,635 RF schools and 7,391 non-RF Title I schools. 

Non-response rate across items, years, and groups: 0 to 12.3 percent. 



Administrative support activities are another key responsibility of reading coaches. Reading 
coaches in RF schools were significantly more likely to report several administrative support 
activities as central to their work than were coaches in non-RF Title I schools, including 
participation in professional development (93 vs. 85 percent), compiling reading assessment data 
(92 vs. 73 percent), and administering or coordinating reading assessments (88 vs. 71 percent) 
(Exhibit 6.3). For both RF and non-RF Title I schools, these findings are consistent with reading 
coach reports in 2004-05. 
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Exhibit 6.3 



Percentage of Reading Coaches’ Rating Various Administrative and Schooi Support Activities as 
Centrai to Their Work in RF and Non-RF Titie I Schoois, 2004-05 and 2006-07 Schooi Years 



2004-05 



2006-07 



Difference in Difference 



*88 



*88 







67 









67 





34 

33 



76 

70 



Administer/coordinate reading 
assessments 
- 3.9 

Compile reading assessment 
data 
- 1.8 

Facilitate or coordinate family 
literacy activities 
- 3.2 

Order/manage reading 
instruction materials 
7.5 



*36 _ 
26 



78 

75 



89 

I 1 1 1 1— 

100 80 60 40 20 



Provide sub time for teachers to 
observe other more experienced 
teachers 
11 . 8 * 

Participate in school leadership 
team meetings 
8.6 

Participate in professional 
development provided by the 
district, state or other consultants 



□ RF Schools 

□ Non-RF Title I Schools 

88 * 

[Ti 



92* 

73 



34 

35 



75* 

61 



39* 

18 





83* 


72 




- 


93’ 




85 


H 1 1 1 1 1 



0 20 40 60 80 100 



Percent of Schools 



Percent of Schools 



Exhibit reads: In 2004-05, 88 percent of reading coaches in RE schools reported that administering/coordinating reading assessments 
was central to their work, compared to 67 percent of reading coaches in non-RF Title I schools, a statistically significant difference (p 
< .05). In 2006-07, 88 percent of reading coaches in RF schools reported that administering/coordinating reading assessments was 
central to their work, compared to 7 1 percent of reading coaches in non-RF Title I schools, a statistically significant difference (p < 
.05). The 2004-05 and 2006-07 results differ by -3.9 percentage points (right-hand column) reflecting the fact that there was a 17 
percentage point difference between RF and non-RF Title I schools in 2006-07 and a 21 percentage point difference in 2004-05. 
Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.6.3 for additional statistics. 

Source: Reading Coach Survey (Question B4 in 2004 -05 and 2006 -07). 

Weight: Reading coach. 

Weighted respondents: Reading coaches in 1,635 RF schools and 7,391 non-RF Title I schools. 

Non-response rate across items, years, and groups: 0 to 12.3 percent. 
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Activities Supporting Teachers’ instruction Undertaken by Reading Coaches 

Reading coaches in RF schools were more likely than coaches in non-RF Title I schools to report 
a variety of coaching activities that support teachers’ reading instruction as central to their work. 
In 2006-07, 85 percent or more of the reading coaches in RF schools rated six of ten coaching 
activities as central to their work (Exhibit 6.4), whereas, no coaching activity was rated as such 
by a comparable proportion of reading coaches in non-RF Title I schools. In particular, RF 
coaches were more likely than coaches in non-RF Title I schools to rate the following activities 
as central to their work: assisting teachers in using the core reading program (89 percent vs. 60 
percent) and observing and providing feedback to teachers (92 percent vs. 64 percent). Of note, 
the two activities rated by the greatest proportion of coaches in non-RF Title I schools as central 
to their work were helping to design strategies for struggling readers (83 percent), and helping to 
monitor effectiveness of strategies for struggling readers (77 percent). This may be because 
these coaches are more like reading specialists than the coaches in RF schools. 

Fewer coaches in non-RF Title I schools rated providing assistance to teachers in using the core 
reading program as central to their work in 2006-07 than in 2004-05 (60 percent vs. 77 percent), 
while in RF schools almost 90 percent of coaches rate this activity as central to their work in 
both time periods. Across other reading coach activities there is little change from 2004-05 to 
2006-07 for both groups. 
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Exhibit 6.4 



Percentage of Reading Coaches Rating Various Activities Supporting Teachers’ instruction as Centrai 
to Their Work in RF and Non-RF Titie i Schoois, 2004-05 and 2006-07 Schooi 



*89 



77 



72 

70 



*79 



56 



"92 



72 



90 

86 



87 

83 



*72 



64 



*88 



69 



*83 



68 



2004-05 



*31 



22 



I 1 1 1 1 

100 80 60 40 20 

Percent of Schools 



Difference in Difference 



Materiais 

Assist teachers in using the core 
reading program 
17.6* 

Give demonstration lessons with 
core/supplemental materials 
17.1* 

Assessment 

Give demonstrations on assessment 
administration/scoring 
-13.3* 

Help teachers in interpreting assessment 
results 
2.9 

Struggiing Readers 

Help teachers design strategies for 
struggling readers 
7.5* 

Help teachers monitor the effectiveness 
of strategies for struggling readers 
11.7* 

Organization/Pianning 

Plan reading instruction with teachers 
11.7* 

Observe and provide feedback to 
teachers 
9.6* 

Assist teachers in forming 
instructional groups 

5.5 

Review teachers’ lesson plans & 
provide feedback 

1.6 



31* 



20 







60 


- 




79* 




60 


- 




72* 




61 



2006-07 



□RF Schools 
□Non-RF Title I Schools 



89* 



97* 



74 



95* 



83 



93* 



77 

77* 



57 



92* 



64 



88 * 



68 



20 40 60 80 

Percent of Schools 



100 



Exhibit reads: In 2004-05, 72 percent of reading coaches in RE schools reported that giving demonstration lessons with 
core/supplemental materials was central to their work, compared to 70 percent of reading coaches in non-RF Title I schools. In 2006- 
07, 79 percent of reading coaches in RF schools reported that giving demonstration lessons with core/supplemental materials was 
central to their work, compared to 60 percent of reading coaches in non-RF Title I schools, a statistically significant difference (p < 
.05). The 2004-05 and 2006-07 results differ by 17.1 percentage points (right-hand column) reflecting the fact that there was a 19 
percentage point difference between RF and non-RF Title I schools in 2006-07 and a 2 percentage point difference in 2004-05. This 
difference in differences was statistically significant (p < .05). 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.6.4 for additional statistics. 

Source: Reading Coach Survey (Question B5 in 2004 -05 and 2006 -07). 

Weight: Reading coach. 

Weighted respondents: Reading coaches in 1,635 RF schools and 7,391 non-RF Title I schools. 

Non-response rate across items, years, and groups: 0 to 12.3 percent. 
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Direct Classroom Support to Teachers for Improving Reading Instruction 

Principals and teachers answered a series of questions about the support teachers receive, beyond 
professional development, to improve their reading instruction. This additional support may be 
in the form of coaching or mentoring by a reading coach or fellow teacher, classroom 
demonstrations, help from a reading coach on intervention services, or help with interpretation of 
assessment data. 

Teachers in RF schools were more likely report to having direct support for reading instruction 
across a variety of topics in than were teachers in non-RF Title I schools, including help 
interpreting assessment data (91 percent vs. 70 percent), using assessment data to determine 
which topics require additional instruction (92 percent vs. 76 percent), and diagnostic testing 
help from a reading coach or specialist (72 percent vs. 48 percent) (Exhibit 6.5). It is also worth 
noting that 50 percent or more of RF teachers reported having the majority of these supports for 
instruction (eight of ten supports). In contrast, for only four of ten support activities did 50 
percent or more of teachers in non-RF Title I schools report having these supports. 

The overall reported levels of direct classroom support provided to teachers changed little from 
2004-05 to 2006-07. 
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Exhibit 6.5 



Direct Support for Reading instruction Received by Teachers in RF and Non-RF Titie I Schoois, 
2004-05 and 2006-07 Schooi Years 



2004-05 



52 

*52 



31 



*57 



47 



*38 



31 



*87 



72 



*89 



*90 



74 



67 



*45 



28 



*85 



100 



— I — 

80 



48 

— I — 

60 



— I — 

40 



— I — 

20 



Percent of Teachers 



Difference in Difference 



*76 








49 












*70 





Struggiing Readers 

Diagnostic testing help from a reading 
coach or specialist for individual students 
- 2.8 

Intervention service help from a reading 
coach or specialist for individual students 
3.2 

Peer-Peer Assistance 

Peer study group for group study 
-2.9 

Coaching from fellow teacher 
-0.5 



Observations of other teachers 
2.6 

Grade level meetings devoted to reading 
1.6 

Assessment 

Using assessment data to determine 
topics that require additional instruction 
1.0 

Interpretation of assessment data 
-1.4 

Coaching 

Demonstrations in my classroom 
0.0 



Coaching by reading coach in 
programs, materials or strategies 
3.2 



2006-07 



□ RF Schools 

□ Non-RF Title I Schools 



48 



72 * 



73* 



34 



52 

52* 

50* 



40 



34* 



25 



74 



76 



91* 



92* 



91* 



70 



42* 



25 



85* 



45 



1 

20 



1 

40 



1 

60 



1 

80 



100 



Percent of Teachers 



Exhibit reads: In 2004-05, 76 percent of teachers in RE schools reported that diagnostic testing help from a reading coach or 
specialist was available in their schools, compared to 49 percent of teachers in non-RF Title I schools. This difference is statistically 
significant (p < .05). In 2006-07, 72 percent of teachers in RF schools reported that this type of direct support was available, 
compared to 48 percent of teachers in non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 
2006-07 results differ by -2.8 percentage points reflecting a 24 percentage point difference in 2006-07 and a 27 percentage point 
difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.6.5 for additional statistics. 

Source: Teacher Survey (Question D2 in 2004-05, Question E2 in 2006-07). 

Weight: School 

Weighted respondents: Teachers in 1,591 RF schools and 13,926 non-RF Title I schools. 

Non-response rate across groups, items, and years: 0.9 to 5.5 percent 
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Teachers were asked to identify the types of direct classroom support not available in their 
schools. A significantly greater percentage of teachers in non-RF Title I schools reported that 
direct supports for reading instruction across all the activities listed were not available than did 
teachers in RF schools (Exhibit 6.6). For example, in 2006-07, teachers in non-RF Title I 
schools were more likely than teachers in RF schools to report that observations of other teachers 
were not available (40 percent vs. 21 percent), that demonstrations conducted in their classrooms 
were not available (46 percent vs. 17 percent) and that help in interpreting assessment data was 
not available (17 percent vs. 2 percent). These same patterns of availability (or lack thereof) 
were reported in 2004-05. 
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Exhibit 6.6 



Direct Support for Reading Instruction Not Available to Teachers in RF and Non-RF Titie I Schoois, 
2004-05 and 2006-07 Schooi Years 



2004-05 



Difference in Difference 



*32 



10 



*29 



38 



*57 



20 



*28 

26 
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*17 



*18 



aL 



*44 
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iJ 


[’ 
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Percent of Teachers 



Struggling Readers 

Diagnostic testing help from a reading 
coach or specialist for individual 
students 
1.9 

Intervention service help from a reading 
coach or specialist for individual 
students 
-1.4 

Peer-Peer Assistance 

Peer study group for group study group 
2.5 

Coaching from fellow teacher 
-4.8* 



Observations of other teachers 
- 8 . 8 * 

Grade level meetings devoted to 
reading 
-2.7 

Assessment 

Using assessment data to determine 
topics that require additional 
instruction 
0.8 

Interpretation of assessment data 
-0.7 

Coaching 

Demonstrations in my classroom 
-4.5* 



Coaching by reading coach in 
programs, materials or strategies 
2.7 
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2006-07 



□ RF Schools 

■ Non-RF Title I Schools 





30* 
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29* 
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32* 








21 
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1 
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Exhibit reads: In 2004-05, 8 percent of teachers in RF schools reported that diagnostic testing help from a reading coach or specialist 
was not available in their schools, compared to 32 percent of teachers in non-RF Title I schools. This difference is statistically 
significant (p < .05). In 2006-07, 8 percent of teachers in RF schools reported that this type of direct support was available, compared 
to 30 percent of teachers in non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 
results differ by 1.98 percentage points reflecting a 22 percentage point difference in 2006-07 and a 24 percentage point difference in 
2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.6.6 for additional statistics. 

Source: Teacher Survey (Question D2 in 2004-05, Question C2 in 2006-07). 

Weight: Teacher 

Weighted respondents: Teachers from 1,591 Reading First schools and 13,926 non-RF Title I schools. 

Non-response rate across groups, years, and items ranges from 0.9 to 5.5 percent. 
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Oversight of Reading Programs 



Oversight of reading program implementation is provided by not only school but also by district 
and state personnel. In the classroom, however, the primary oversight comes from schools’ 
principals. In this section, we present findings about oversight provided at the principal, district, 
and state levels. 

Responsibility for Oversight of Reading Programs 

Principals in RF and non-RF Title I schools identified those entities responsible for supporting 
and providing oversight of reading activities (i.e. state, district, principal, or reading coach). 
According to principals, the states have a greater oversight role in RF schools than in non-RF 
Title I schools, including monitoring the implementation of the reading program (43 percent vs. 
7 percent), interpretation of assessment results (26 percent vs. 12 percent) and selection of 
professional development topics (20 percent vs. 5 percent) in 2006-07 (Exhibit 6.7). 
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Exhibit 6.7 



Responsibility for Reading Activities in RF and Non-RF Title I Schools, 2004-05 and 2006-07 
School Years 
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-07 


2006-07 


Responsible Party 


RF 
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Non-RF 
Title 1 
Schools 


RF 
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Non-RF 
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Difference 
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...School alone is responsible 
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-0.2 


...School, district and state are jointly responsible 


23* 


13 


26* 


12 


4.2 


. . .Other combinations 


0 


4* 


0 


2 


2.3* 


Review individuai students’ progress in 
reading 












...School alone is responsible 
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Exhibit reads: In 2004-05, 33 percent of principals in RF schools reported that the school alone was responsible for 
monitoring the implementation of the reading program, compared to 50 percent of principals in non-RF Title I schools, a 
statistically significant difference (p < .05). In 2006-07, 32 percent of principals in RF schools reported that the school alone 
was responsible for monitoring the implementation of the reading program, compared to 53 percent of principals in non-RF 
Title I schools, a statistically significant difference (p < .05). The 2004-05 and 2006-07 results differ by -4.7 percentage 
points (right-hand column) reflecting the fact that there was a -21 percentage point difference between RF and non-RF Title I 
schools in 2006-07 and a -16 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.6.7 for additional statistics. 

Source: Principal Survey (Question D5 in 2004-05 and D3 in 2006-07). 

Weight: Principal. 

Weighted respondents: Principals in 1,684 RF schools and 14,684 non-RF Title I schools. 

Non-response rate across items, years, and groups: < 1 percent. 
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Role of the School Principal in the Evaluation of K-3 Reading Instruction 

Most principals (more than 90 percent) in RF and non-RF Title 1 schools reported observing their 
K-3 teachers’ reading instruetion, either formally or informally (Exhibit 6.8a). However, RF 
principals were more likely to informally observe teaehers more frequently (at least monthly) 
than were principals in non-RF Title I sehools (78 pereent vs. 64 percent). Also while almost all 
principals in both RF and non-RF Title 1 schools meet with teaehers to diseuss strategies to 
improve reading instruetion, prineipals in RF schools reported having such meetings more 
frequently, either individually (at least monthly, 49 pereent vs. 40 percent), or in groups (at least 
monthly, 59 percent vs. 49 percent) than prineipals in non-RF Title 1 schools (Exhibit 6.8b). 



Exhibit 6.8a 



Percentage of RF and Non-RF Title I Schools in Which K-3 Reading Instruction Was 
Evaluated by the Principal, by Mode and Frequency of Evaluation, 2004-05 and 2006-07 
School Years 
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instruction informally 

■ ...Once a week or more 
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□ ...Not at all 

Exhibit reads: In 2004-05, 64 percent of principals in RF schools reported that they observed classroom reading 
instruction informally (at least weekly), compared to 46 percent of principals coaches in non-RF Title I schools. This 
difference is statistically significant (p < .05). In 2006-07, 64 percent of principals in RF schools reported conducting this 
activity at least weekly, compared to 5 1 percent of principals in non-RF Title I schools. The 2004-05 and 2006-07 results 
differ by 3 percentage points reflecting the fact that there was a 15 percentage point difference between RF and non-RF 
Title I schools in 2006-07 and an 18 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.6.8 for additional statistics. 

Source: Principal Survey (Question D6 in 2004-05 and D4 in 2006-07). 

Weight: Principal. 

Weighted respondents: Principals in 1,684 RF schools and 14,684 non-RF Title I schools. 

Non-response rate across items, years, and groups: < 1 percent. 



Observed classroom reading 
instruction using an evaiuation form 
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Exhibit 6.8b 



Percentage of RF and Non-RF Title I Schools in Which Principal Met with Teachers to 
Discuss Strategies to Support Instruction, by Mode and Frequency of Evaluation, 2004-05 
and 2006-07 School Years 
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Exhibit reads: In 2004-05, 19 percent of principals in RF schools reported that met with teachers individually (at least 
weekly) to discuss strategies for improving reading instruction, compared to 46 percent of principals coaches in non-RF 
Title I schools. This difference is not statistically significant. In 2006-07, 21 percent of principals in RF schools reported 
conducting this activity at least weekly, compared to 15 percent of principals in non-RF Title I schools. This difference is 
statistically significant (p < .05). The 2004-05 and 2006-07 results differ by 1 percentage point reflecting the fact that 
there was a 5 percentage point difference between RF and non-RF Title I schools in 2006-07 and a 4 percentage point 
difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.6.8 for additional statistics. 

Source: Principal Survey (Question D6 in 2004-05 and D4 in 2006-07). 

Weight: Principal. 

Weighted respondents: Principals in 1,684 RF schools and 14,684 non-RF Title I schools. 

Non-response rate across items, years, and groups: < 1 percent. 



88 



Chapter 6: Oversight and Classroom Support Activities 


























Summary 



Both in terms of classroom support and oversight, teachers in RF schools appear to have more 
support for reading instruction than do teachers in non-RF Title I schools. Reading coaches 
represent an important source of support for teachers, and RF schools were significantly more 
likely to have a reading coach than were non-RF Title I schools (99 percent vs. 57 percent). 
Furthermore, 75 percent of reading coaches in RF schools reported that they are full-time reading 
coaches, compared to only 19 percent in non-RF Title I schools, which allows teachers in RF 
schools to benefit from substantially more in-school support. 

Reading coaches in RF schools were significantly more likely to rate a variety of instructional 
and administrative support activities as central to their work than were coaches in non-RF Title I 
schools. The following activities were rated as such by a greater proportion of reading coaches 
in RF schools than in non-RF Title I schools: assisting teachers in using the core program (89 
percent vs. 60 percent); forming instructional groups (88 percent vs. 68 percent); monitoring the 
effectiveness of strategies for struggling readers (93 percent vs. 77 percent); compiling reading 
assessment data (92 percent vs. 73 percent); and ordering or managing reading instruction 
materials (75 percent vs. 61 percent). In addition, RF teachers were more likely to report 
receiving ongoing, direct support for teaching reading than were teachers in non-RF Title I 
schools, such as: interpretation of assessment data (91 percent vs. 70 percent), assistance from a 
reading coach or specialist in diagnosing individual student needs (72 percent vs. 48 percent) or 
intervention service help for individual students (73 percent vs. 52 percent). 

Principals and district and state personnel also provide support and oversight for reading 
instruction. States were more likely to be involved in RF schools than in non-RF Title I schools; 
a greater proportion of RF principals reported state involvement across multiple activities, 
including monitoring the implementation of the reading program (43 percent vs. 7 percent), 
interpretation of assessment results (26 percent vs. 12 percent) and selection of professional 
development topics (20 percent vs. 5 percent). Principals are also important sources of 
oversight, and virtually all principals (more than 90 percent) in RF and non-RF Title I schools 
reported that they observe their teachers’ reading instruction and hold meetings with teachers. 
However, principals in RF schools were more likely to report that they evaluate reading 
instruction informally at least monthly than principals in non-RF Title I schools (78 percent vs. 

64 percent). 
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Chapter?: Professional Development 



One of the aims of the Reading First program is to strengthen teachers’ knowledge and 
understanding about teaching reading such that they can more effectively teach and provide 
support to their students. Professional development is one of the mechanisms used to provide 
teachers with the tools they need to do so. An important provision of the RF legislation is that 
professional development be available to staff in all schools, not only those with RF funding. 
Specifically, K-3 teachers are eligible to participate in professional development paid for by 
district RF funds, and K-12 Special Education teachers are eligible to participate in professional 
development paid for by state RF funds. 

The Reading First program deliberately funds professional development activities to train 
teachers on teaching methods that scientifically based reading research has shown to be effective. 
Professional development that fosters evidence-based teaching practices reinforces teachers’ 
understanding of reading education such that they can effectively support learning in students in 
the early grades, particularly those children who are struggling academically. 

This chapter summarizes findings on three aspects of professional development activities, 
including: a) their structure and design features (e.g., workshops, conferences, and the 
availability of stipends, required attendance); b) activities related to the five dimensions of 
reading instruction emphasized by Reading First; and c) activities related to other features of 
reading instruction (e.g., instructional strategies for grouping students, using materials and 
assessments). 

Key Findings 

A greater percentage of RF teachers reported participating in reading-related professional 
development activities in the 2006-07 school year than did teachers in non-RF Title I 
schools (90 percent vs. 73 percent). On average, teachers in RF schools reported participating 
in twice the number of workshops (4.9 vs. 2.3) and spending twice the number of hours (31.3 
hours vs. 15.7 hours) in professional development activities than did teachers in non-RF Title 1 
schools in the previous year. However, the mean number of workshops and number of hours 
participating in reading-related professional development declined for both teachers in RF and 
non-RF Title 1 schools from 2004-05 to 2006-07. 

The proportion of teachers in both RF and non-RF Title I schools who reported 
participating in professional development in three key dimensions of reading 
(comprehension, phonics and phonemic awareness) was unchanged from 2004-05 to 2006- 
07. In both years, teachers in RF schools were more likely than teachers in non-RF Title I 
schools to report participating in professional development activities on these topics. For 
vocabulary and fluency, however, the percentage of teachers in non-RF Title 1 schools who 
reported attending professional development increased from 2004-05 to 2006-07 (from 67 
percent to 74 percent on fluency, and from 5 1 percent to 60 percent on vocabulary). For teachers 
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in RF schools, there was little change in their reporting of participation in professional 
development on these two dimensions of reading. 

A greater proportion of staff in RF schools than in non-RF Title I schools reported 
participating in professional development activities addressing other features of reading 
instruction (i.e., other than the five dimensions of reading) that provided new information 
about effective teaching strategies to use during reading instruction (e.g., grouping, 
assessment, struggling readers) in the previous year. RF teachers were more likely to report 
receiving professional development in providing assistance to struggling readers (85 percent vs. 
71 percent), how to use assessment to inform instruction (91 percent vs. 81 percent), and how to 
group students (84 percent vs. 74 percent) than were teachers in non-RF Title 1 schools. Even 
though most teachers in both types of schools had some professional development to help 
struggling readers, almost three-quarters of teachers in both RF and non-RF Title I schools 
reported needing additional professional development on this topic. 

Professional Development to Improve Reading Instruction 

The Reading First legislation specifies that professional development activities should provide 
teachers with “information on instructional materials, programs and approaches based on 
scientifically based reading research” (Section 1202). The text below discusses the structure and 
design features of professional development in the five dimensions of reading emphasized by the 
Reading First program as well as professional development on other features of reading 
instruction (i.e., other than the five dimensions of reading) including teaching strategies, 
grouping, assessment, evaluation), and direct classroom support to improve reading instruction. 

The Structure of Professional Development Activities for Teachers 

In 2006-07, a greater percentage of teachers in RF schools reported participating in reading- 
related professional development activities than did teachers in non-RF Title I schools (90 
percent vs. 73 percent). For example, they were more likely to attend fiill-day workshops (63 
percent vs. 42 percent) and conferences (66 percent vs. 51 percent) related to reading than were 
teachers in non-RF Title I schools (Exhibit 7.1). Further, teachers in RF schools reported 
participating in twice the number of workshops (4.9 vs. 2.3) and spending twice the number of 
hours in professional development activities (31.3 hours vs. 15.7 hours). 
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Exhibit 7.1 



Types of Reading-Related Professional Development Activities for Teachers in RF and Non-RF 
Title I Schools, 2004-05 and 2006-07 School Years 
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Difference 
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Difference 


Half-day workshops 
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66%* 


51% 


1.3% 


Full-day workshops 


80* 


50 


63* 


42 


-8.8* 


Any workshop 


97* 


85 


90* 


73 


5.8* 


College courses 


19* 


14 


15* 


11 


-0.3 


Conferences 


36* 


27 


29* 


17 


2.4 


Number of Workshops 


7.0* 


Mean 

3.3 


4.9* 


Mean 

2.3 


-1.1* 


Hours attending reading-related professional 
development activities 


41.5* 


17.6 


31.3* 


15.7 


-8.4* 



Exhibit reads: In 2004-05, 74 percent of teachers in RE schools reported attending a half-day workshop, compared to 61 
percent of teachers in non-RF Title I schools. This difference is statistically significant (p < .05). In 2006-07, 66 percent of 
teachers in RF schools reported attending a half-day workshop, compared to 5 1 percent in of teachers in non-RF Title I schools. 
This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by 1 percentage point (right-hand 
column) reflecting a 15 percentage point difference between RF and non-RF Title I teachers in 2006-07 and a 14 percentage 
point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and 
non-RF Title I schools. 

See Appendix B, Exhibit B.7.1 for additional statistics. 

Source: Teacher Survey (Question D1 in 2004-05 and Cl in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,684 RF schools and 14,656 non-RF Title I schools. 

Non-response rate across groups, years, and items ranges from < 1 percent. 



However, the mean number of workshops and number of hours participating in reading-related 
professional development declined for both teachers in RF and non-RF Title I schools from 
2004-05 to 2006-07. For RF schools, this could indicate that there was a greater need for 
professional development at the outset of the grant. 

The Reading First program specifically provides funding for states to support professional 
development activities that include teachers in schools that did not receive RF funding provided 
those schools serve low-achieving students. Teachers in non-RF Title I schools that are in RF 
districts reported participating in significantly more workshops than their peers in non-RF Title I 
schools that are in districts that did not receive RF funds (3.4 vs. 2.5) and reported participating 
in significantly more hours of professional development (18.4 hours vs. 14.1 hours). 
Nevertheless, teachers in RF schools reported, on average, participating in significantly more 
workshops and participating in more hours of professional development than either group of 
teachers in non-RF Title I schools, whether in an RF or a non-RF district (5.5 workshops and 
31.3 hours). This indicates that while professional development activities might be available to 
teachers in non-RF Title I schools statewide, those teachers in schools in RF districts (in both RF 
and non-RF Title I schools) were more likely to attend than their counterparts in districts that did 
not receive RF funds. 
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Specific Design Features of Professionai Deveiopment 

In 2006-07, teachers in RF schools were significantly more likely to report that the professional 
development activities they participated in were led by trainers with well-established reputations 
(78 percent vs. 64 percent), were held in a convenient location (72 percent vs. 61 percent), used a 
team-based approach (69 percent vs. 53 percent), and provided follow-up activities (46 percent 
vs. 34 percent) than their peers in non-RF Title I schools (Exhibit 7.2). Further, teachers in RF 
schools were significantly more likely to report being offered incentives to participate in 
professional development, such as release time (44 percent vs. 33 percent) and stipends (37 
percent vs. 20 percent) than teachers in non-RF Title I schools. Professional development 
activities with these characteristics are more likely to be meaningful learning and training 
opportunities (Corcoran, 1995; Caret et al, 1999; Learning First Alliance, 2003). 
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Exhibit 7.2 



Design Features of the Professionai Deveiopment Activities for Teachers in RF and in Non-RF Titie I 
Schools, 2004-05 and 2006-07 School Years 



2004-05 



2006-07 




Difference in Difference 




□ RF Schools 




-l 


□ Non-RF Title 1 Schools 



Are given by trainers or facilitators who 
have a well-established reputation 
- 0.1 

Are held in a convenient location (e.g., 
activities held at school) 

- 2.2 

Use a team-based approach Qoint 
training of people who work together) 
-3.7* 



Were also attended by the principal 
-5.7* 



Provide follow-up activities 
-4.6* 



Include release time for participating 
teachers 
- 2.2 



Provide a stipend 
- 8 . 6 * 



Provide teachers options among which 
to choose 
1.9 

Offer graduate college credits 
- 2.6 



78* 



64 



12 ^ 



61 



69* 



53 

52* 



34 



44 

46* 

44* 



33 

37* 



20 



33 

33 



21 ^ 

17 



20 40 60 80 100 



Percent of Activities^ 



Percent of Activities 



Exhibit reads: In 2004-05, RE teachers reported that 78 percent of the professional development activities offered to them is given by 
trainers or facilitators who have a well-established reputation, compared to 63 percent reported by teachers in non-RF Title I schools. 
This difference is statistically significant (p < .05). In 2006-07, RF teachers reported that 78 percent of the professional development 
activities offered to them is given by trainers or facilitators who have a well-established reputation, compared to 64 percent reported by 
teachers in non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by -0.1 
percentage points, reflecting a 14 percentage point difference in 2006-07 and a 15 percentage point difference in 2004-05. 

^Response choices were ‘none,’ ‘one-quarter,’ ‘one-half,’ ‘three-quarters’ and ‘all.’ Each response was converted to a corresponding 
percentage (0, 25, 50, 75 and 100 percent, respectively). 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.7.2 for additional statistics. 

Source: Teacher Survey (Question D3 in 2004-05 and C3 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,684 RF schools and 14,656 non-RF Title I schools. 

Non-response rate across groups, years, and items is < 1 percent. 
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Professional Development on the Five Dimensions of Reading 

Scientifically based reading research focuses attention on phonemic awareness, phonics, 
vocabulary development, fluency, and comprehension. Successful implementation of the 
elements of the reading program can be greatly bolstered by the availability of effective 
professional development opportunities in the five dimensions. 

Teachers ’ Participation in Professional Development on the Five Dimensions of Reading 
The proportion of teachers in both RF and non-RF Title I schools who reported participating in 
professional development on three of the five key dimensions of reading (comprehension, 
phonics and phonemic awareness) was unchanged from 2004—05 to 2006-07. For example, in 
both years, over 87 percent of teachers in RF schools and almost 75 percent of teachers in non- 
RF Title I schools participated in professional development on comprehension (Exhibit 7.3). For 
vocabulary, however, the percentage of teachers in non-RF Title I schools who reported 
participating in professional development increased from 5 1 percent in 2004-05 to 60 percent 
2006-07. 

As was evident in 2004-05, teachers in RF schools continue to be much more likely than 
teachers in non-RF Title I schools to report participating in professional development on the five 
key dimensions of reading in 2006-07: comprehension (88 percent vs. 74 percent), phonics (88 
percent vs. 64 percent), phonemic awareness (87 percent vs. 62 percent), vocabulary (82 percent 
vs. 60 percent), and fluency (91 percent vs. 74 percent). 

As mentioned above, teachers in non-RF Title I schools in RF districts were more likely than 
their peers in non-RF districts to reported having attended professional development workshops. 
However, there were no differences between these two groups of teachers in non-RF Title I 
schools on their reports of participating in professional development activities on the five key 
dimensions of reading. 
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Exhibit 7.3 



Teacher Participation in Professionai Deveiopment Activities Reiated to the Five Dimensions of 
Reading: RF and Non-RF Titie I Schoois, 2004-05 and 2006-07 Schooi Years 
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Exhibit reads: In 2004-05, 87 percent of teachers in RE schools reported participating in professional development on 
comprehension, compared to 73 percent of teachers in non-RF Title I schools. This difference is statistically significant (p < .05). In 
2006-07, 88 percent of teachers in RF schools reported participating in this type of professional development, compared to 74 
percent of teachers in non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results 
differ by 0.7 percentage points (center column) reflecting a 13.8 percentage point difference in 2006-07 and a 14.5 percentage point 
difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.7.3 for additional statistics. 

Source: Teacher Survey (Question D4 in 2004-05 and C4 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,684 RF schools and 14,629 non-RF Title I schools. 

Non-response rate across groups, years, and items: 0.1 to 10.8 percent. 
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Teachers also reported on their perceived professional development needs in the five dimensions 
of reading. In 2006-07, teachers in RF schools were less likely than teachers in non-RF Title I 
schools to report needing additional professional development in phonemic awareness (28 
percent vs. 48 percent), phonics (43 percent vs. 56 percent), and fluency (38 percent vs. 49 
percent) (Exhibit 7.4). In addition, fewer teachers in both RF and non-RF Title I schools reported 
the need for professional development in fluency in 2006-07 than in 2004-05. Teachers in RF 
schools also were less likely to report needing professional development on comprehension, 
phonics and phonemic awareness in 2006-07 than were their counterparts in 2004-05. 



Exhibit 7.4 

Teachers’ Perceived Professionai Deveiopment Needs Reiated to the Five Dimensions of 
Reading: RF and Non-RF Titie i Schoois, 2004-05 and 2006-07 Schooi Years 
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Exhibit reads: In 2004-05, 73 percent of teachers in RE schools reported a perceived need related to comprehension professional 
development, compared to 72 percent of teachers in non-RF Title I schools. This difference is not statistically significant. In 
2006-07, 65 percent of teachers in RF schools reported this perceived need, compared to 71 percent of teachers in non-RF Title I 
schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by -7.6 percentage points 
reflecting a 1.2 percentage point difference in 2006-07 and a -6.4 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non- 
RF Title I schools. 

See Appendix B, Exhibit B.7.4 for additional statistics. 

Source: Teacher Survey (Question D4 in 2004-05 and C4 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,673 RF schools and 14,575 non-RF Title I schools. 

Non-response rate across groups, years, and items ranges from 0.5 to 11.1 percent. 



□ RF Schools 

□ Non-RF Title I Schools 



Teacher Ratings of Their Preparedness to Teach the Five Dimensions of Reading 
The goal of professional development in the five dimensions of reading is to strengthen teaehers’ 
knowledge of these eomponent skills, thereby improving their teaehing of reading. In 2006-07, 
RF teaehers reported they were better prepared than teaehers in non-RF Title I sehools to teaeh 
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all five dimensions of reading (on a scale of one to five, with five representing extremely well, 
Exhibit 7.5). For example, in 2006-07, RF teachers, on average, rated their preparedness to 
teach phonemic awareness a 4.1, while teachers in non-RF Title I schools, on average, rated their 
preparedness a 3.5. These findings are similar to the 2004-05 findings; on phonemic awareness, 
for example, RF teachers, on average rated themselves at 3.9 compared to an average of 3.3 for 
teachers in non-RF Title I schools. 



Exhibit 7.5 



Preparedness to Teach Five Dimensions of Reading: Teachers in RF and Non-RF Titie I Schoois, 
2004-05 and 2006-07 Schooi Years 
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Exhibit reads: In 2004-05, RE teachers reported a mean preparedness score of 3.9 to teach phonemic awareness, compared to 3.3 
in teachers in non-RF Title I schools. This difference is statistically significant (p < .05). In 2006-07, teachers in RF schools 
reported a mean preparedness score of 4.1 to teach phonemic awareness, compared to an average of 3.5 for teachers in non-RF 
Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by 0 points (right- 
hand column), reflecting a 0.6 point difference in 2006-07 and a 0.6 point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non- 
RF Title I schools. 

See Appendix B, Exhibit B.7.5 for additional statistics. 

Source: Teacher Survey (Question D6 in 2004-05 and C6 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,684 RF schools and 14,656 non-RF Title I schools. 

Non-response rate across groups, years, and items ranges from 0 to 0.5 percent. 



Principals^ and Reading Coaches^ Participation in Professional Development in the Five Dimensions 
of Reading 

In 2006-07, a significantly greater percentage of principals in RF schools reported participating 
in professional development on all five dimensions of reading than principals of non-RF Title I 
schools (e.g., fluency: 84 percent vs. 65 percent; phonics: 78 percent vs. 57 percent) (Exhibit 
7.6). However, from 2004-05 to 2006-07, there were modest decreases in RF principals’ 
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attendance at professional development on these topics (e.g., phonemic awareness from 89 
percent to 79 percent). Over the same time period, there was little change reported by principals 
of non-RF Title I schools’ attendance at professional development on these topics. 



Exhibit 7.6 



Professionai Deveiopment Reiated to the Five Dimensions of Reading Attended by for Principais and 
Reading Coaches in RF and Non-RF Titie I Schoois, 2004-05 and 2006-07 Schooi Years 
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Exhibit reads: In 2004-05, 88 percent of principals in RF schools reported participating in vocabulary professional development, 
compared to 63 percent of principals in non-RF Title I schools. This difference is statistically significant (p < .05). In 2006-07, 84 
percent of principals in RF schools reported participating in this type of professional development, compared to 69 percent of 
principals in non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by - 
10.5 percentage points reflecting a 14.9 percentage point difference in 2006-07 and a 25.4 percentage point difference in 2004-05. 
Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.7.6 for additional statistics. 

Source: Principal Survey (Questions F2 in 2004-05 and FI in 2006-07); Reading Coach Survey (Question G in 2004-05 and FI in 
2006-07). 

Weight: School. 

Weighted respondents: Principals or reading coaches in 1,634 RF schools and 14,456 non-RF Title schools. 

Non-response rate across groups, years, and items ranges from 0 to 3.0 percent. 



In 2006-07, the percentage of RF reading coaches who reported participating in professional 
development related to the five dimensions of reading was significantly greater than reading 
coaches in non-RF Title I schools (83 percent vs. 68 percent). From 2004-05 to 2006-07, the 
percentage of RF reading coaches who reported participating in such professional development 
declined from 94 to 83 percent, while the percentage of reading coaches in non-RF Title I 
schools increased from 60 to 68 percent. 
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Professional Development on Other Features of Reading Instruction 

In addition professional development in the five dimensions of reading, teachers, principals and 
reading coaches were asked about professional development related to other features of reading 
instruction. We categorized scientifically based reading instruction (SBRI)-related professional 
development activities into five areas: 1) using materials and teaching strategies; 2) grouping; 3) 
assessment of students; 4) how to help struggling readers; and 5) organizing and planning (See 
Appendix D for the specific activities included in each of these topics). 

Teachers ’ Participation 

To summarize teachers’ reported professional development, we calculated a score representing 
the proportion of teachers who reported participating in professional development in any topic in 
one of the five categories listed above. We then used this summary measure to compare the 
professional development activities of RF teachers and teachers in non-RF Title I schools in 
2004-05 and 2006-07 (Exhibit 7.7). Teachers in RF schools were more likely to report 
participating in professional development aimed at helping struggling readers (85 percent vs. 71 
percent), in teaching strategies (96 percent vs. 90 percent), using assessment to inform 
instruction (91 percent vs. 81 percent), and grouping students for reading instruction (84 percent 
vs. 74 percent) than were teachers in non-RF Title I schools. However, it is worth noting that 
almost three-quarters of teachers in non-RF Title I schools reported participating in all five 
categories of professional development related to other features of reading instruction. 
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Exhibit 7.7 



Professionai Deveiopment Reiated to Other Features of Reading Instruction for Teachers in RF and 
Non-RF Titie I Schoois, 2004-05 and 2006-07 Schooi Years 



2004-05 



*98 



90 

*90 



72 



83 

81 

*80 
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68 



65 



Difference in Difference 
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Area of Professional 
Development 
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Assessment 

-IT 
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- 3.4 
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□ RF Schools 
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90 
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Exhibit reads: In 2004-05, 98 percent of teachers in RE schools reported participating in teaching strategies professional 
development, compared to 90 percent of teachers in non-RF Title I schools. This difference is statistically significant (p < .05). In 
2006-07, 96 percent of teachers in RF schools reported participating in this type of professional development, compared to 90 
percent of teachers in non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results 
differ by -1.9 percentage points (right-hand column) reflecting a 6.3 percentage point difference in 2006-07 and an 8.2 percentage 
point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.7.7 for additional statistics. 

Source: Teacher Survey (Question D5 in 2004-05 and C5 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,684 RF schools, and 14,575 non-RF Title I schools. 

Non-response rate across groups, years, and items ranges from 0.5 to 1.6 percent. 



We also asked teachers about their perceived need for additional professional development 
related to other features of reading instruction (Exhibit 7.8). In 2006-07, there were few 
differences between teachers in RF and non-RF Title I schools. The two topics about which 
most teachers (75 percent or more) in both RF and non-RF Title I schools reported needing more 
professional development were in general teaching strategies and working with struggling 
readers. Approximately half (or fewer) teachers reported needing more professional 
development about other categories. 
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Exhibit 7.8 



Teachers’ Perceived Need for Additionai Professionai Development Related to Other Features of 
Reading Instruction in RF and non-RF Title I Schools, 2004-05 and 2006-07 School Years 
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Exhibit reads: In 2004-05, 78 percent of teachers in RE schools reported needing additional professional development in teaching 
strategies, compared to 74 percent of teachers in non-RF Title I schools. This difference is statistically significant (p < .05). In 
2006-07, 81 percent of teachers in RF schools and non-RF Title I schools reported needing more of this type of professional 
development. The 2004-05 and 2006-07 results differ by -3.5 percentage points (center column) reflecting a -0.2 percentage point 
difference in 2006-07 and a 3.4 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between teachers in RF and non-RF 
Title I schools. 

See Appendix B, Exhibit B.7.8 for additional statistics. 

Source: Teacher Survey (Question D5 in 2004-05 and C5 in 2006-07). 

Weight: School. 

Weighted respondents: Teachers in 1,684 RF schools and 14,629 non-RF Title I schools. 

Non-response rate across groups, years, and items ranges from 0.2 to 2.0 percent. 



Principals^ Participation in Professional Development 

Principals were asked to report on the professional development they had received in the 
previous calendar year as well as on their perceived needs for professional development. In all 
categories but organization and planning, a significantly greater percentage of RF principals 
reported participating in professional development about other features of reading instruction 
than principals in non-RF Title I schools (e.g., evaluation 80 percent vs. 61 percent) (Exhibit 
7.9). Principals’ reported participation in professional development on these topics have not 
changed from 2004-05 to 2006-07. 
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Exhibit 7.9 



Professionai Deveiopment Reiated to Other Features of Reading instruction Attended by Principais 
in RF and Non-RF Title I Schools, 2004-05 and 2006-07 School Years 
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Exhibit reads: In 2004-05, 93 percent of principals in RF schools reported participating in professional development in teaching 
strategies, compared to 81 percent of principals in non-RF Title I schools. This difference is statistically significant (p < .05). In 
2006-07, 90 percent of principals in RF schools reported this type of professional development, compared to 84 percent of 
principals in non-RF Title I schools. This difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ 
by -5 percentage points reflecting a 6 percentage point difference in 2006-07 and a 12 percentage point difference in 2004-05. 
Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.7.9 for additional statistics. 

Source: Principal Survey (Question F2 in 2004-05 and FI in 2006-07). 

Weight: Principal. 

Weighted respondents: Principals in 1,684 RF schools and 14,684 non-RF Title I schools. 

Non-response rate across groups, items, and years: 0 to 1.8 percent 



Reading Coaches^ Participation in Professional Development 

Reading coaches provide ongoing professional development to teachers in two ways: by 
modeling effective instructional practice and by supporting teachers’ efforts to implement their 
reading programs. The guidance for the Reading First Program states that “delivery mechanisms 
[for professional development] should include the use of coaches and other teachers of reading 
who provide feedback as instructional strategies are put into practice’’ (2002, page 26). 

Therefore, it is critical that coaches receive the support and training in the tenets if SBRI, how to 
effectively guide and provide feedback to teachers, and how model high quality classroom 
teaching. 
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A significantly greater percentage of RF reading coaches reported participating in professional 
development across almost all topics than reading coaches in non-RF Title I schools (Exhibit 
7.10). For example, RF reading coaches reported participating in more professional 
development on assessment topics than did coaches in non-RF Title I schools (e.g., types of 
assessments: 81 percent vs. 63 percent). In addition, coaches in RF schools were more likely to 
report attending professional development on the essential components of scientifically based 
reading instruction (83 percent vs. 68 percent), and explicit reading instruction (83 percent vs. 59 
percent) than were coaches in non-RF Title I schools. 

Reading First coaches reported decreased participation in two professional development topics 
from 2004-05 to 2006-07: essential components of SBRI (from 94 percent to 83 percent) and 
the role of the reading coach in fostering change (from 84 percent to 73 percent). These finding 
may be due to the fact that since these are introductory topics and this sample is composed of 
mature RF schools, coaches had already participated in professional development on these 
topics. Reading coaches also reported increased participation over the same time period on how 
to plan instructional interventions for struggling students (from 73 percent to 83 percent). 

From 2004-05 to 2006-07, reading coaches in non-RF Title I schools reported increased 
participation in professional development on several topics related to teaching strategies 
including, for example, planning instructional interventions (from 57 percent to 68 percent), 
helping teachers identify appropriate instructional materials (from 43 percent to 55 percent) 
using assessment data to form instructional groups (from 68 percent to 77 percent) and classroom 
management within the literacy block (from 45 percent to 55 percent). 
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Exhibit 7.10 



Professional Development in Teaching Strategies Attended by Reading Coaches in RF and Non- 
RF Title I Schools, 2004-05 and 2006-07 School Years 
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05 


2006- 


-07 


2006-07 


Topics 


RF 

Schools 


Non-RF 
Title 1 
Schools 


RF 

Schools 


Non-RF 
Title 1 
Schools 


Difference 
- 2004-05 
Difference 


Scientifically Based Reading Instruction 






Essential components of scientifically based 
reading instruction 


94%* 


60% 


83%* 


68% 


-18.7% 


How to plan instructional interventions for 
struggling students 


73* 


57 


83* 


68 


-1.0 


How to help teachers make reading instruction 
systematic and explicit 


79* 


52 


83* 


59 


-2.9 


Assessment 






How to use reading assessment data to guide 
instruction 


95%* 


78% 


93%* 


83% 


-6.4 


What are the types of assessments: screening, 
diagnostic, progress monitoring, and outcome 


88* 


54 


81* 


63 


-15.7* 


How to use assessment data to form 
instructional groups 
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68 


91* 


77 


-9.0* 


How to Provide Effective Support and Feedback to Teachers 






What is the role of the reading coach in 
fostering change 


84* 


51 


73* 


54 


-13.6* 


How to provide constructive feedback to 
teachers 
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60 


77* 


61 


-4.8 


How to establish credibility with teachers 


73* 


41 


69* 


52 


-14.3* 


Classroom management within the literacy 
block time 


61* 


45 


68* 


55 


-3.1 


How to conduct effective grade level meetings 


57* 


33 


66* 


46 


-3.8 


How to help teachers identify appropriate 
instructional materials 


CD 


43 


67* 


55 


-8.2 


How to provide onsite professional 
development 


70* 


50 


74* 


55 


-2.2 


Effective Modeling 






How to conduct demonstration lessons 


61* 


51 


63 


55 


-2.7 


How to conduct classroom observations 


75* 


49 


76* 


54 


-3.0 



Exhibit reads: In 2004-05, 94 percent of reading coaches in RE schools reported participating in professional development about 
essential components of scientifically based reading instruction, compared to 60 percent of reading coaches in non-RF Title I 
schools. This difference is statistically significant (p < .05). In 2006-07, 83 percent of reading coaches in RF schools reported 
participating in this type of professional development, compared to 68 percent of reading coaches in non-RF Title I schools. This 
difference is statistically significant (p < .05). The 2004-05 and 2006-07 results differ by -19 percentage points (right-hand 
column) reflecting a 15 percentage point difference in 2006-07 and a 34 percentage point difference in 2004-05. 

Note: The stars (*) indicate that there is a statistically significant difference (i.e., p-value < .05) between RF and non-RF Title I 
schools. 

See Appendix B, Exhibit B.7.10 for additional statistics. 

Source: Reading Coach Survey (Question G1 in 2004-05 and FI in 2006-07). 

Weight: Reading coach. 

Weighted respondents: Reading coaches in 1,635 RF school and 7,667 non-RF Title I schools. 

Non-response rate across groups, items, and years: < 1 percent. 
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Summary 



Data about professional development for teachers, principals, and reading coaches from the 
2006-07 school year, in combination with data from 2004-05, provide some evidence that 
participation in Reading First-aligned professional development activities continues, and that 
teachers in both RF and non-RF Title I schools are increasingly prepared to provide sound 
reading instruction. 

A greater percentage of RF teachers reported participating in reading-related professional 
development than did teachers in non-RF Title I schools in 2006-07. On average, compared to 
teachers in non-RF Title I schools, RF teachers reported attending twice the number of 
workshops and spending twice the number of hours in professional development activities. 
However, the mean number of workshops and number of hours attending reading-related 
professional development activities declined for both teachers in RF and non-RF Title I schools 
from 2004-05 to 2006-07. 

In 2006-07, teachers in RF schools were significantly more likely to report that the professional 
development activities they participated in were led by trainers with well-established reputations, 
were held in a convenient location , used a team-based approach, and provided follow-up 
activities than their peers in non-RF Title I schools. 

The proportion of teachers in both RF and non-RF Title I schools who reported participating in 
professional development in three of the five key dimensions of reading (comprehension, 
phonics, phonemic awareness and fluency) was unchanged from 2004-05 to 2006-07. In both 
years, teachers in RF schools were more likely than teachers in non-RF Title I schools to report 
attending professional development activities on these topics. For vocabulary and fluency, 
however, the percentage of teachers in non-RF Title I schools who reported attending 
professional development increased from 2004-05 to 2006-07. 

A greater percentage of teachers in RF schools reported participating in other reading-related 
professional development directed at helping teachers with teaching strategies grounded in 
scientifically based reading research (e.g., assistance to struggling readers, interpretation of 
assessment data, grouping students) compared to their non-RF Title I counterparts. There were 
few differences, however, in teachers’ perceived need for additional professional development 
on other features of reading instruction with the exception of interpreting assessment results, 
about which teachers in non-RF Title I schools reported a greater need than RF teachers. In both 
groups of teachers, at least three-quarters of RF and non-RF Title I teachers reported a need for 
additional professional development to assist struggling readers. A higher percentage of 
principals and reading coaches in RF schools also reported participating in professional 
development related to other features of reading instruction, relative to their counterparts in non- 
RF Title I schools. 
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Chapter 8: Reading Achievement Trends in RF and 
Non-RF Title I Schoois 



The overarching goal of the RF program is to promote reading skills, such that all students are 
reading at or above grade level by the end of third grade. RF is predicated on scientifically 
researched findings that high-quality reading instruction in the primary grades significantly 
reduces the number of students who experience reading difficulties in later years. Reading 
proficiency by the end of third grade, in turn, establishes a necessary foundation for successful 
performance across a broad range of skills and competencies in later grades. The following 
analyses examine the extent to which reading achievement is improving more quickly in RF 
schools than in non-RF Title I schools. 

This chapter presents findings from two analyses of reading achievement trends in RF and non- 
RF Title I schools and is organized into two sections: The first presents analyses of third- and 
fourth-grade students’ performance on state reading assessments to address evaluation question 
4: Does student achievement improve over time more quickly in schools with Reading First 
funds than in non-RF Title I schools not receiving RF funds?^^ The second section describes an 
analysis of the relationship between schools’ implementation of RF-aligned activities and student 
reading achievement to address question 5 of the evaluation: Is there any relationship between 
how schools implement Reading First-aligned practices and changes in reading achievement? 

Key Findings 

There is limited but statistically significant evidence that successive cohorts of third- and 
fourth-grade students in RF schools improved their reading performance over time more 
quickly than did their counterparts in non-RF Title I schools. 

On third-grade state reading assessments, effect sizes averaged across 24 states indicate that RF 
schools gained more on their state third-grade reading assessments, on average, from pre- to 
post-RF implementation than non-RF Title I schools, a statistically significant yet small 
difference (p < .001 for all four methods). In addition to calculating average effect sizes, we 
conducted separate state-by-state analyses of reading performance. In 12 of 24 states, the 
improvement in third-grade reading scores among RF schools was statistically significantly 
The footer should say chapter 8: Reading Achievement Trends in RF and non-RF Title I Schools 
larger than in non-RF Title I schools for at least one of four methods; in the other 12 states, there 



This chapter relies on reading assessment data collected from individual states. In general, most states use these 
state assessments to meet the accountability requirements of NCLB. These assessments are not used in most 
cases to meet RF accountability requirements. 

^ To conduct the state -by- state analyses we had to identify the test scores that represented schools’ performance 
in years prior to as well as in years after RF implementation. Because the dates when RF schools received 
funding were not available, we used four different methods to estimate these dates, and in turn, to identify 
schools’ pre- and post-RF implementation years. 
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were no statistically significant differences between the two groups of schools in any of the 
methods. 

On fourth- grade state reading assessments, effect sizes averaged across 17 states indicate that RF 
schools gained more on their state fourth-grade reading assessments, on average, from pre- to 
post-RF implementation than non-RF Title I schools, also a statistically significant yet small 
difference (p < .01 across all four methods). Across the state-by-state analyses, in six of 17 
states the improvement in fourth-grade reading scores among RF schools was statistically 
significantly larger than in non-RF Title I schools for at least one of the four methods. In 1 1 of 
17 states there were no significant differences in improvement between the two types of schools. 

It is important to note the findings about student reading performance should be interpreted with 
caution. Although the RF and non-RF Title I schools are similar demographically on several 
characteristics measured in the study, schools were not randomly assigned to receive RF funding, 
and therefore there could have been preexisting differences between the two groups of schools 
on unobserved characteristics. We cannot assume that the two groups of schools are equivalent 
but for the fact that one group received RF funds while the other did not. This means that the 
findings cannot support causal inferences that attribute observed differences in student reading 
achievement between RF and non-RF Title I schools to the Reading First program.^' 

There is a positive and statistically significant relationship between only one of four 
measures of RF-aligned activities and schools’ levels of third-grade reading achievement.^^ 

The study team analyzed the relationship between schools’ third-grade reading scores on state 
assessments and four composite measures constructed from survey data that characterize 
teachers’ RF-aligned activities: classroom reading instruction; strategies to help struggling 
readers; participation in professional development; and uses of assessment to inform instruction. 
The four composites were each constructed from selected survey responses of teachers from the 
spring 2005 survey administration. The analysis was conducted on a group of 390 schools, all 
part of the original study sample of 1,633 schools; these schools had both survey responses and 
third-grade scores on their states’ reading assessments. 

Only the composite measure of teachers’ use of activities for struggling readers was significantly 
related to the probability that a school scored in the top quartile (relative to other RF and non-RF 
Title I schools) on their state’s third-grade reading assessment. For every increase of one 
standard deviation unit in the struggling readers implementation composite score, the probability 



Additional limitations to the analysis of student reading achievement are presented in the discussion section of 
this chapter. 

® This analysis includes both RF and non-RF Title I schools, since the implementation of activities aligned with 
Reading First can potentially occur in either type of school. 
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of being in the top quartile increased by 15.6 percentage points, for the average school (p < 
.001).“ 

Reading Performance of Third- and Fourth-Grade Students on State 
Reading Assessments 

Overview of Data and Analytic Approach Used in the Analyses 

Below, we briefly describe the data and analytic methods used to compare reading performance 
trends in RF and non-RF Title 1 schools. More detailed descriptions of the statistical models 
used in the analyses are provided in Appendix C. 

Data 

Analyses were conducted using annual third- (24 states) and fourth-grade (17 states) test scores 
on state reading assessments from 12,362 schools (3,000 RF and 9,362 non-RF Title 1). The 
analyses are based on state reading assessment data from as early as 1997 and as recent as 2006, 
from RF and non-RF Title I schools, located in 35 states (See Appendix B, Exhibit B.8.1 for 
details about the number and years of test scores used in each state analysis). The assessments 
vary widely across states, and school-level scores are reported in a variety of metrics (Exhibit 
8.1). The most common metric is the percentage of students meeting some cutoff score, such as 
’’basic” or ’’proficient” level, as defined by individual states. These data are available in national 
databases for all schools; schools were included in the present analysis if they were regular 
schools (private, vocational, alternative, and special education schools were excluded), either an 
RF or a non-RF Title 1 school.^ 



The analysis of the relationship among RF-aligned activities and reading achievement is based on fitting a 
logistic regression model because the dependent variable is dichotomous-inclusion or exclusion from the 
highest quartile. The four composites were standardized and are best understood in terms of standard deviation 
units. 

^ The analyses of student reading performance includes Title I schoolwide project (SWP) schools. That is, 
schools in which at least 40 percent of the students are eligible for free or reduced price lunches and have 
elected to implement programs where all children are eligible for Title I programs and services. 



Chapter 8: Reading Achievement Trends in RF and Non-RF Title I Schools 



111 





Exhibit 8.1 



Summary of Data Used for Anaiysis of State Reading Assessment Scores, by State 
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Exhibit 8.1 



Summary of Data Used for Anaiysis of State Reading Assessment Scores, by State (Continued) 

Number of 
Schools ^ 

Non-RF Reporting 

Assessment pp jjtle | Metric 



Mississippi ^ 




^b 


Mississippi Curriculum Test 


31 


325 


Percent students at or 
above proficient 


Montana 






Iowa Test of Basic Skills 


16 


45 


Percent students nearing 
proficient or above 


Nevada 






Criterion Referenced 
Reading Test 


26 


55 


Percent students at or 
above Level 2 
(approaching standard) 


New Jersey 






N.J. Assessment of Skills 
and Knowledge 


49 


178 


Percent student at or 
above proficient 


North Carolina 






End of Grade Test 


87 


463 


Percent students at or 
above consistent mastery 


North Dakota 






North Dakota State 
Assessment Program 


12 


17 


Percent students meeting 
or exceeding standard 


Ohio 




^b 


Ohio Achievement Test 


75 


639 


Percent students at or 
above proficient 


Oregon 






Oregon Statewide 
Assessment 


34 


154 


Percent students at or 
above standard 


South Carolina 






South Carolina Palmetto 
Achievement Challenge 
Test 


49 


276 


Percent students at or 
above basic level 


South Dakota 






South Dakota State Test on 
Educational Progress 


14 


52 


Percent students at or 
above proficient 


Utah 






Utah Performance 
Assessment System of 
Students 


12 


49 


Percent students at or 
above basic 


Virginia 






Virginia Standards of 
Learning 


70 


186 


Percent students at or 
above standard 


Washington 






Iowa Test of Basic Skills 
(Grade 3), Washington 
Assessment of Student 
Learning (Grade 4) 


63 


223 


ITBS: Percent students at 
or above the 50th 
percentile; WASL: percent 
students at or above Level 
2 (basic) 


West Virginia 






West Virginia Educational 
Standards Test 


31 


249 


Percent students scoring 
at or above proficient 
(previously known as 
mastery) 


Wisconsin 


^b 




Wisconsin Reading 


49 


133 


Percent students at or 



Comprehension Test above basic level (both 

(Grade 3), Wisconsin WRCT and WKCE) 

Knowledge and 
Concepts Exam (Grade 4) 

Exhibit reads: In Arizona, data were available for analysis of third-grade reading aehievement based on students’ performanee 
on the Arizona Instrument to Measure Standards; seores were reported as the pereentage of students reaehing the standard. 
Test seore data were available for 67 RF sehools and 290 non-RF Title I sehools. 

^ Minimum number of sehools for any analysis. 

^ There was insuffieient pre- or post-RF implementation years (and seores) to eonduet analyses using the state modeling 
strategy. 

Souree: Databases from 35 states, eaeh of whieh ineludes sehool-level measures of reading performanee on the states’ third- 
and fourth-grade reading assessments 



Grades 
Included in 
Analysis 

3rd 4th 

State grade grade 
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To conduct these analyses, we identified, for each school, those years (and their corresponding 
test scores) that occurred both before and after the implementation of RF. There is no one data 
source that tells us when each RF school received RF funds. However, using information that 
tells us when states received their grants, when RF districts received their subgrants, and when 
the study sample of RF schools reported receiving RF funds (reported on the principal survey), 
we developed four different methods to assign pre- and post-RF implementation years to all 
schools included in the analyses. Each of these methods is described below.^^ 

• Initial State Award Date 

Test scores in years prior to the state award date were designated as pre-RF 
implementation years. Scores after the last school in the state received funding are 
designated as post-RF implementation years. This allows sufficient time for the funds 
to have been distributed to all of the schools. Scores from the intervening years, when 
some schools may have been funded while others were not yet funded, are excluded from 
the analysis. In most cases, this method resulted in two years of data being excluded 
from the analyses. 

• District Award Date 

This method used the district award dates provided by the U.S. Department of Education 
to assign award dates to all RF schools. The award dates were then used to specify pre- 
and post-RF implementation years. One problem with this method is that, based on 
information from our surveys, there is often a time lag between districts’ receipt of funds 
and schools’ receipt of funds. 

• Adjusted District Award Date 

For this method, we followed the approach described above but added the average 
number of days between a district award and a school receiving funds to the district 
award date to account for the time that had elapsed. We used the information reported by 
principals in their surveys to estimate an average time gap between district and school 
receipt of funding and adjusted the district award date by adding to it an average number 

f\l 

of days to account for this time gap. 

• School Award Date 

In this approach, we used the school funding dates provided on RF principals’ surveys, 
and proportionally imputed dates to other RF schools and non-RF schools in the RF 
districts in our study sample. For schools in the remaining districts, we continued to use 
the adjusted award date described above. 



Procedures used to assign schools’ award dates and corresponding pre-and post RF implementation years are 
presented in Appendix C. 

The last school funded was designated using from the Southwest Educational Development Laboratory (SEDL) 
Reading First Awards Database as of May 2005. Schools that received funds after this date would not have had 
adequate test score data to be included in these analyses; specifically, their first post-RF implementation year 
would be the 2006-07 school year. Test score data for that year were not available when we conducted these 
analyses. 

Survey data to compute the average state level time gap were available for all states. 
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To conduct the four analyses based on the methods described above, in some instances, we 
imputed start dates in order to identify pre- and post-RF implementation years. The extent to 
which such dates were imputed varied depending on the analytic method. For instance, the 
analysis based initial state award dates required no imputation because this method defines one 
start date for the onset of RF and this date was applied to all RF and non-RF Title I schools in the 
analysis. Also, for the analyses using RF district award dates, there was no imputation for the RF 
and non-RF Title I schools in those districts since those dates define the onset of RF 
implementation. However, 60 percent of the non-RF Title I schools included in the analyses are 
in districts with no RF schools and therefore, no RF district start dates. For those schools, we 
used dates from RF districts to impute schools’ start dates. (A more detailed description of the 
methods used to assign start dates is presented in Appendix C.) 

Analytic Approach 

Difference-in-difference models were fit to the data to estimate whether the changes in reading 
performance in RF schools were larger than in non-RF Title I schools. Depending on data 
availability, for each state, either three or four separate models were fit to the data. In most 
states, we fit baseline mean models which estimated the average reading performance in the pre- 
RF years in both RF and non-RF Title I schools, compared it to the post-RF average 
performance, and then compared the non-RF Title I pre-post difference to the RF pre-post 
difference (i.e., the difference-in-difference estimate). 

In states that had at least five years of data available during the pre-RF years and at least two 
years of data during the post-RF years, we fit linear models to the data (i.e., nine states at third 
grade, none at fourth grade). These model estimated the expected average school-level reading 
performance in the absence of RF in both RF and non-RF Title I schools, given the trajectory of 
performance in the pre-RF implementation years. The expected reading performance was then 
compared to the actual performance in RF and non-RF Title I schools to estimate differences 
within RF and non-RF Title I schools. These models also compared the difference between the 
actual and expected performance across RF and non-RF Title I schools (i.e., the difference-in- 
difference estimate). 

We combined estimates across states by converting each estimate into an effect size, a 
standardized measure that describes the magnitude of the original difference in terms of the 
standard deviation of the original metric. These state level effect sizes were then weighted and 
averaged across states for each model (Shadish and Haddock, 1994).^^ Describing the findings 
across states in terms of effect sizes allows us to include all states with appropriate test data even 
though the metrics used to report reading performance differ. One consequence, however, is that 



We used the standard deviation of Title I schools pre-RF implementation scores in the states’ effect size 
calculations. See Exhibits B.8.2b and B.8.3b in Appendix B for individual state effect sizes. 

® The weight was inversely proportional to the variance of the effect size resulting in more precise effect sizes 
carrying more weight than less precise effect sizes. 
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effect sizes can be somewhat more difficult to interpret than more familiar metrics/*' Therefore, 
we converted the average effect sizes for each of the four analytic methods into a percentage 

7 1 

point difference, because this is the original metric most often used by states. 

Reading Achievement in Third Grade 

Analyses of scores on states’ third- and fourth-grade state reading assessments provide 
limited evidence that students in RF schools improved their reading performance more 
quickly than did their counterparts in non-RF Title I schools. 

Results are presented first for third grade, because this is the grade level for which we have data 
from the most states. It is also at the end of third grade that Reading First aims to have children 
reading on or above grade level. This is followed by a discussion of findings for fourth grade. 

Exhibit 8.2 presents the results of analyses of third-grade test scores in 24 states; each column 
provides the difference-in-difference estimate converted to an effect size for the four models that 
represent different definitions of the start of RF implementation. Using the smallest school count 
across the four methods for each state, these results represent approximately 186,000 students in 
more than 2,200 RF schools, corresponding to about 55 percent of third- grade RF students, 
nationwide. 

States are listed in roughly descending order based on the magnitude of differences in reading 
gains between RF and non-RF Title I sehools. In 1 1 states (in order of magnitude from the 
analysis using state award dates: Mississippi, Virginia, South Dakota, Oregon, West Virginia, 
Washington, Illinois, North Carolina, California, Arizona, and Maryland), the gains in RF 
sehools were statistieally signifieantly larger than the gains in non-RF Title I sehools, for three 
out of the four methods. Further, in seven of the states, the differenees are statistieally 
signifieant for all four methods. In 12 states, there was no signifieant differenee between the 
estimated gains in RF and non-RF Title I sehools. 



Research on interpreting effect sizes suggests that it is unwise to rely on universal guidelines to interpret the 
magnitude of an effect (Hill, Bloom, Black and Lipsey, 2007). More recent thinking advises developing 
empirical benchmarks of the substantive significance of an effect size, based on the particular field of research 
and population being studied. 

To convert effect sizes into a percentage point differences, we multiplied them by the pooled standard deviation 
for non-RF Title I schools across states that use a cut-score as the metric used to report student reading 
performance. This percentage point difference reflects differences across all states in the analysis, even those 
that did not originally report reading achievement in this metric. 

In 1 5 states, we fit baseline mean models for all four methods to estimate differences in third-grade performance 
over time between RF and non-RF Title I schools; in eight states, we fit linear models (shaded in this exhibit) 
for all four methods', and in one state, Illinois, linear models were fit for all but the state award date method, for 
which the data required us to fit a baseline mean model. 
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Exhibit 8.2 

Difference-in-Difference Effect Size Estimates for Four Analytic Methods, in RF and Non-RF 
Title I Schools' Third-Grade Reading Scores 



Difference-in-Difference Estimates 



state 


state Award 
Date 


District 
Award Date 


Adjusted District 
Award Date 


School 
Award Date 


Mississippi 


1.07* 


1.10* 


1.12* 


1.12* 


Virginia 


0.88* 


0.87* 


0.87* 


0.87* 


South Dakota 




0.41* 


0.42* 


0.42* 


Oregon 


0.64* 


0.69* 


0.48* 


0.48* 


West Virginia 




0.41* 


0.42* 


0.46* 


Washington 


0.52* 


0.40* 


0.34* 


0.40* 


Illinois 


0.16* 


0.36* 


0.24* 


0.24* 


North Carolina 


0.40* 


0.25* 


0.23* 


0.22* 


California ^ 




0.27* 


0.21* 


0.13* 


Arizona 


0.41* 


0.51* 


0.40* 


0.37* 


Maryland 


0.32- 


0.40* 


0.37* 


0.38* 


South Carolina 


0.43* 


0.10 


0.09 


0.09 


Georgia 


0.12 


-0.46 


0.50 


0.59 


Indiana 


-0.01 


0.10 


0.09 


0.09 


Massachusetts 


0.14 


0.02 


0.03 


0.05 


Florida 


-0.04 


-0.02 


-0.05 


-0.06 


Kentucky 


0.13 


-0.31 


-0.28 


-0.21 


Hawaii 


0.02 


-0.24- 


-0.24- 


0.08 


Colorado 


0.02 


-0.16 


-0.17 


-0.17 


Wisconsin 




-0.06 


-0.06 


-0.22 


District of Columbia 




-0.07 


-0.07 


-0.07 


Nevada 


-0.22 


-0.18 


-0.19 


-0.23 


Minnesota 


-0.06 


-0.02 


-0.02 


-0.02 


Delaware 


-0.45 


-0.44 


-0.45 


-0.45 


Average Effect Size 


.21* 


.18* 


.18* 


.17* 



Exhibit reads: In Mississippi, when pre- and post-RE implementation years were defined using the state’s award date, 
the differenee in differenee analysis produeed an effeet size of 1.07 standard deviations indieating that from pre- to post- 
RE implementation the gain in the pereentage of students at or above profieient was greater in RE sehools than in non- 
RE Title I sehools. This differenee is statistieally signifieant (p < .05). 

^ California had data from the California Standards Test (CST), an English or language arts test that measures students’ 
reading and writing skills. Beeause the CST measures multiple skills, rather than reading alone, results from this test are 
not ineluded in this exhibit. However, an analysis of CST generated results that are similar to the results reported above 
for the CAT-6; third-grade students in RF sehools made small but statistieally signifieant greater gains in performanee 
than did students in non-RE Title I sehools. 

Note: Shaded eells indieate that there were adequate data to support an analysis using a linear model; for all other states 
the analysis refleets the use of a baseline-mean model. The analysis using the linear analyses eompares aetual average 
student aehievement to what the linear model prediets we would have expeeted in the absenee of Reading First. The 
data for Illinois were adequate for fitting linear models in all but the state award date model. 

Statistieally signifieant differenees between RE and non-RE Title I sehools are indieated using the following symbols: 
p-value < .10 = and p-value < .05 = *. 

See Appendix B, Exhibit B.8.2 for more detailed state-level statisties ineluding differenee in differenee estimates 
presented in their original metries. 
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Averaging effect sizes across the 24 states indicate that RF schools gained more, on average, 
from pre-to post-RF implementation than non-RF Title I schools on the states’ third- grade 
reading assessments. The average effect size ranged from .17-.21 standard deviations, a 
statistically significant yet small difference (p < .001 for all four methods). A chi-square test 
comparing the average effect sizes across the four methods indicate that the findings are not 
statistically significantly different from one another. These effect sizes are equivalent to small 
differences, ranging from 2.4 to 3.0 percentage point differences in the percentage of students 
meeting their states’ proficiency standards. 

Reading Achievement in Fourth Grade 

The analysis of fourth-grade scores allows us to examine whether students’ exposure to Reading 
First in earlier grades leads to gains in their reading performance as they progress into higher 
grades. Fourth-grade test scores were examined in the 17 states where fourth-grade students 
were in third grade during the implementation of Reading First. In six states (Florida, 

Wisconsin, Colorado, Mississippi, New Jersey, and Washington) the gain in reading 
performance was statistically significantly larger in RF schools than in non-RF Title I schools for 
at least one of four methods (Exhibit 8.3). In four of these states (Florida, Mississippi, 
Washington, and Wisconsin), the improvement was statistically significant for all methods. In 
Colorado and New Jersey only one of the four methods was statistically significant at the 0.05 
level. In 1 1 states, there were no significant differences in gains between RF and non-RF Title I 
schools in any of the four methods. 

To estimate the pre- to post-change in fourth-grade reading performance across states, we pooled 
the reading scores across states by calculating a weighted average effect size for each of the four 
methods. RF schools gained statistically significantly more from pre-to post-RF implementation 
than non-RF Title I schools; the effect size ranged from .17-.24 across the four methods (p < .01 
across all four methods). A chi-square test comparing average effect sizes across the four 
methods indicate that results are not statistically significantly different from each other. 

Converting the effect sizes into percentage point differences, they correspond to 2. 0-2. 9 
percentage points, indicating that while statistically significant, these differences are small. That 
is, across states RF school gains correspond to between two and three percentage points more 
from pre- to post-RF implementation than non-RF Title I schools in the proportion of students 
meeting proficiency on their fourth-grade reading assessments. 



The post-RF implementation years included in the fourth-grade analyses are 2004-05 and 2005-06; these 
fourth-grade students experienced RF as third-grade students in 2003-04 and 2004-05 respectively. 
Information from the SEDL database indicates that the RF schools included in these analyses were still 
participating in RF when these students were in third grade. 



118 



Chapter 8: Reading Achievement Trends in RF and Non-RF Titie i Schoois 





Exhibit 8.3 

Difference-in-Difference Effect Size Estimates for Four Methods, in RF and Non-RF Title I 
Schools' Fourth-Grade Reading Scores 

Difference-in-Difference Estimates 

Adjusted 



State 


state Award 
Date 


District 
Award Date 


District 
Award Date 


School 
Award Date 


Florida 




0.48* 


0.31* 


0.31* 


Wisconsin 


0.40* 


0.28* 


0.28* 


0.30* 


Colorado 




0.37* 


0.24- 


0.21 


Mississippi 




0.58* 


0.59* 


0.60* 


New Jersey 


0.24- 


0.15 


0.17 


0.25* 


Washington 


0.47* 


0.45* 


0.34* 


0.42* 


Connecticut 


0.25 


0.29- 


0.29- 


0.16 


Montana 


0.32 


0.27 


0.27 


0.27 


Louisiana 


0.18- 


0.11 


0.16- 


0.17- 


Utah 




0.14 


0.14 


0.14 


North Dakota 


0.07 


0.01 


0.01 


0.01 


Michigan 




0.09 


0.07 


0.08 


Kentucky 


0.05 


0.06 


-0.08 


0.00 


Idaho 


-0.10 


0.12 


0.12 


0.11 


Arkansas 




-0.03 


-0.03 


-0.03 


Iowa 


-0.09 


-0.14 


-0.14 


-0.14 


Ohio 




-0.12 


-0.12 


-0.12 


Average Effect Size 


.20* 


.24* 


.17* 


.18* 



Exhibit reads: In Wisconsin, when pre- and post-RE implementation years were defined using the state’s award date, 
the difference in difference analysis produced an effect size of .40 standard deviations indicating that the gain from 
pre- to post-RF implementation in the percentage of students at or above basic level was greater in RF schools than in 
non-RF Title I schools. This difference is statistically significant (p < .05). 

Note: Statistically significant differences between RF and non-RF Title I schools are indicated using the following 
symbols: p-value < .10 = and p-value < .05 = *. 

See Appendix B, Exhibit B.8.3 for more detailed state-level statistics including difference in difference estimates 
presented in their original metrics. 



Discussion 

In both third- and fourth-grade analyses of student reading performance, the estimates of the 
differences in reading achievement pre- and post-RF in both RF and non-RF Title I schools are 
quite consistent across the four analytic methods. That is, the estimates for a state were often 
very close across methods and the tests for statistical significance portrayed a consistent story. 
Further, the tests for statistical significance were consistent across methods in 22 states. In 
fourth grade, the tests for statistical significance were consistent in 13 of 17 states. Finally, the 
average effect sizes are similar across the four methods. For both the third- and fourth-grade 
analyses; the findings from the four methods are not statistically significantly different from one 
another. 



Despite the fact that the findings are consistent across the four methods, this analysis of student 
reading performance should be interpreted with caution in light of several limitations. Although 
the RF and non-RF Title I schools are similar demographically on several characteristics 
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measured in the study, schools were not randomly assigned to receive RF funding, and therefore 
there could have been preexisting differences between the two groups of schools on unobserved 
characteristics. We cannot assume that the two groups of schools are equivalent but for the fact 
that one group received RF funds while the other did not. This means that the findings cannot 
support causal inferences that attribute observed differences in student reading achievement 
between RF and non-RF Title I schools to the Reading First program. 

Second, year-to-year comparisons of performance are based on different cohorts of children, who 
may or may not be comparable over time. Therefore, changes in performance could reflect 
individual student differences rather than the exposure to Reading First. 

Third, most states report reading scores in terms of the percent of students who meet a particular 
cutoff or reference point (i.e., percent at or above basic level), not in terms of each school’s 
average test score). Such proficiency scores mask any student level changes in performance that 
may have occurred above or below the cutoff point. This is particularly worrisome in low- 
performing schools and for low-performing students; even substantial changes in student 
achievement can go undetected when average student reading performance does not cross a 
given threshold. Data on the school-level percentage of students meeting a threshold measure 
may, therefore, underestimate actual improvements in student performance. 

Fourth, because we did not have accurate data on when each school began to implement RF, in 
some instances, we imputed start dates to order to identify schools’ pre- and post-RF 
implementation years. The analysis that used states’ initial award dates required no imputation 
because this method defines one start date for the onset of RF and this date is applied to all RF 
and non-RF Title I schools in the analysis. However, the analyses that used the dates RF districts 
received their funds required imputation for the non-RF Title I schools in districts with no RF 
schools and therefore, no RF district start dates; this represented 60 percent of the non-RF Title I 
school sample. Further, for the analysis based on the start dates of RF schools in our sample, we 
imputed dates for all schools not in RF sample districts; this represented 75 percent of the RF 
schools and 70 percent of the non-RF Title I schools in the analysis. Because these analyses 
required a considerable amount of imputation, it may have led us to misclassify pre- and post-RF 
years for some schools, and consequently may affected the results of these analyses. 

Fifth, state tests typically provide global measures of students’ overall reading performance, 
most often through reading comprehension measures. They do not measure such reading sub- 
skills as phonemic awareness, phonics, or vocabulary that are the focus of evidence-based 
classroom instruction and Reading First. 

Sixth, these analyses are based on states’ average school-level reading assessment scores, and 
because the number of schools varies widely from state to state, this poses an analytic problem. 
California, for example, has approximately 800 RF schools and nearly 100,000 K-3 students, 
whereas Delaware has eight RF schools and fewer than 1,000 K-3 students. The ability to detect 
statistically significant differences decreases as the number of schools within a state decreases. 
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In addition to these limitations, there are three potential counter-arguments that could be raised 
about whether these positive findings are indeed genuine. That is, the results could be influenced 
by 1) regression to the mean, 2) the Pygmalion effect, and 3) ceiling effects in several states. 

Each is discussed in more detail below. 

Regression to the Mean 

RE schools in the majority of states (25 of 35) were performing at a lower level than their non- 
RF Title I counterparts before the implementation of Reading First. It is possible, therefore, that 
the RF schools’ gains relative to non-RF Title I schools are simply an artifact of regression to the 
mean. If the RF schools’ performance levels were originally low simply by chance, one would 
expect their scores to rise in subsequent years not because of Reading First but because higher 
scores are actually characteristic of their true performance. However, 21 of the 35 states 
included in these analyses had three or more years of pre-RF low performance, which is not 
likely to have occurred by chance. In fact, states generally identified RF-eligibility on the basis 
of historical poor academic performance over one or more years. Therefore, absent the RF 
program, we would not expect such schools’ performance to improve because of regression to 
the mean but rather to continue unchanged. We tested this assumption empirically, by estimating 
a correlation between the RF-Title I pre-score difference and the RF gain for each of the four 
methods in both grades. If regression to the mean had occurred, we would expect that the largest 
gains in RF schools would be in those states with the largest pre-RF differences between RF and 
non-RF Title I schools. The estimated correlations in third grade range from .19 to .46 and in 
fourth grade range from .10 to .31, and they are not significant at the .05 level. These 
correlations may be large enough to suggest that perhaps some of the observed differences 
include gains that reflect regression to the mean. 

Pygmalion Effect 

The results raise the question about whether the observed gains may simply be the result of 
Reading First being a new program, which may have the effect of energizing and mobilizing 
staff in their implementation efforts (i.e., the Pygmalion effect). We cannot determine whether 
this is indeed the case. If positive results are sustained over time, then it less likely that they are 
being driven by the Pygmalion effect. Until we have additional post-RF data, however, we 
cannot fiilly address this question (Rosenthal and Jacobson, 1968). 

Ceiling Effect 

When scores on a baseline measure are high, there is a potential ceiling effect, because there is 
little room for gains on that measure posttest. Third-grade baseline scores in 14 of 21 states 
indicate that 70 percent or more of students in non-RF Title I schools scored at or above their 
states’ proficiency thresholds. In five of the 14 states (Mississippi, Virginia, South Carolina, 
Washington, and North Carolina), the proportion of RF schools’ third-graders who met their 
states’ standard was approximately 10 percentage points lower (See Appendix B, Exhibit B.8.2). 
In four of the five states (all but South Carolina), there were statistically significant positive 
findings for the RF schools. For example, in Mississippi, 77 percent of third-grade non-RF 
Title I students scored at or above proficient at baseline, compared to 61 percent of their RF 
peers. There was more opportunity for RF schools’ students to cross the proficiency threshold 



Chapter 8: Reading Achievement Trends in RF and Non-RF Titie i Schoois 



121 





than was the case for their non-RF Title I counterparts. At least in some states then, the gains in 
student reading achievement observed in RF schools may reflect a ceiling effect rather than 
schools’ participation in Reading First. 

Reading Achievement and the Impiementation of RF-aligned Activities 

The primary mechanism through which the Reading First program is expected to affect positive 
changes in student achievement is by promoting the use of scientifically based reading 
instructional practices in the classroom. The program offers support to states (and districts and 
schools) so that they will provide opportunities for high quality professional development and 
research-based curricular materials, with the ultimate goal of improving student reading 
achievement. The analysis presented below examines whether reports of the implementation of 
research-based practices on surveys is related to differences in student achievement within states. 

Overview of Data and Analytic Approach Used in the Analyses 

Below, we provide a brief overview of the data and analytic methods used to examine the 
relationship between reports of Reading First-aligned practices and reading achievement. A 
more detailed description is presented in Appendix C. 

Data 

To investigate the relationship between student reading achievement and the implementation of 
RF activities, we rely on two data sources: 1) third-grade reading scores on state assessments (as 
used above); and 2) selected responses from the spring 2005 teacher survey administration from 
which we constructed four composite variables to characterize key aspects of teachers’ reading 
programs: classroom reading instruction, help for struggling readers, professional development, 
and the use of assessments. 

The sample of schools included in the analysis is drawn from the overall study sample of 1,633 
schools: It includes both RF and non-RF Title I schools, because the implementation of 
activities aligned with Reading First can potentially occur in either type of school. Our analysis 
included only those schools that had teacher survey responses and were located in states with 
sufficient state test score data (n=831). Because tests and metrics vary across states, each state’s 
reading scores were standardized by designating schools as either high- or low-performing 
relative to all other RF and Title I SWP schools in that state.’"^ Selecting schools from only the 
highest and lowest quartiles yielded a final sample of 390 schools from 21 states. 

Analytic Approach 

This analysis explores relationships between teachers’ implementation of RF- aligned activities 
and schools’ performance on states’ third-grade reading assessments. We fit a logistic regression 
model using the four composites to predict the probability that a school scored in the top quartile 

For each state, we selected schools in the lowest quartile (at or below the 25th percentile) and the highest 
quartile (at or above the 75th percentile) on the state’s third-grade reading assessment relative to all RF and 
Title I SWP schools. 
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(of all RF and non-RF Title I schools) on their state’s third-grade reading assessment. The 
model also included indicator variables for each state, in order to account for state level variation 
in the probability of a school’s being in the top quartile on the reading assessment and an 
indicator for Reading First or non-RF Title I status. 

Student Reading Achievement and Impiementation of RF-aiigned Activities 

A logistic regression model was fit to the data to predict a school’s probability of scoring in the 
top quartile (relative to other RF and non-RF Title I schools in their state) using four 
implementation composites as explanatory variables. Only one composite, activities for 
struggling readers, was statistically significantly related to the probability that a school is in the 
top quartile on its state reading assessment (.69, p < .001) (Exhibit 8.4). This finding can be 
explained in terms of its marginal effect on the probability that a school would be in the top 
quartile. The marginal effect of this composite was .156, which means that for the average 
school in our sample, as the composite score increased by one standard deviation, the probability 



Exhibit 8.4 



Using Composite Measures of Impiementation to Predict the Probabiiity of Schoois Scoring 



in the Top Quartile on States’ Reading Tests 


Parameter 


Estimate 


Standard 

Error 


Marginal 

Effect 


P-value 


Intercept 


1.789 


.731 




.015 


Reading First Status 


-2.077 


.352 


-.473 


<.001 


Composite Measures of 










Implementation 










Reading Instruction 


-.270 


.210 


-.061 


.202 


Struggling Readers 


.690 


.162 


.156 


<.001 


P D Activities 


.180 


.166 


.041 


.278 


Uses of Assessment 


-.091 


.167 


-.021 


.587 



Exhibit reads: The estimated relationship between the composite measure of reading instruction and the 
probability of being in the top quartile on the state reading assessment is -.270, with a standard error of .210. 
The marginal effect indicates that for the average school in our sample, as the composite score increases by on 
standard deviation, the probability of being in the top quartile decreases by 6.1 percentage points; this finding 
is not significant at the .05 level. 

Note: The model also included a series of 20 dichotomous state variables (i.e., dummies) to control for state- 
level variation in the probability of a school’s scoring in the top quartile of reading achievement. 

See Appendix C, pages C-9-1 1 for additional details on the variables used to construct the four composite 
measures. 

Sources: Teacher Survey (2004-05); Databases from 21 states that had school-level scores on third-grade 
reading assessments. 
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of being in the top quartile increased by 15.6 percentage points.’^ Also, the marginal effect of 
being an RF school was -.47, meaning that for the average RF school, the probability of being in 
the highest quartile was 47 percentage points less than for non-RF Title 1 schools. (See Exhibit 
8.5, the vertical line at mean=0; the probabilities are .23 for RF schools and .70 for non-RF 
Title I schools.) 

Exhibit 8.5 illustrates the relationship between the predicted probabilities that schools would 
score in the top quartile (the y-axis) and their composite scores characterizing their activities for 
struggling readers (the x-axis). The solid line represents RF, and the dotted line represents non- 
RF Title I schools. The three vertical lines represent composite scores at the one standard 
deviation below the mean score (0), the mean score, one and standard deviation above the mean 
score. 



Exhibit 8.5 

Using Composite Scores on Activities for Struggling Readers to Predict the Probability of 
Schools Scoring in the Top Quartile on States’ Reading Tests 




RF Schools Non-RF Title I Schools 



Exhibit Reads: At one standard deviation below the mean score (vertical line=-l) the predicted probability of 
being in the top quartile is about .12 for RF schools (solid line) and .55 for non-RF Title I schools (dotted 
line). 

Sources; Teacher Survey (2004-05); Databases from 21 states that had school-level scores on third-grade 
reading assessments. 



In this model, the marginal effect of a specific variable is defined as the instantaneous rate of change in the 
probability of a school’s scoring in the top quartile when all composite measures are held constant at their mean 
values. Because the relationship between schools’ scores on the struggling reader composite and the probability 
of scoring in the top quartile is curvilinear (See Exhibit 8.5), the marginal effect of 15.6 would be slightly more 
or less than this amount as schools’ scores on this composite deviate from the mean. 
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There are two findings of note. First, regardless of composite score, RF schools were less likely 
to be in the top quartile than were non-RF Title I schools. Second, across all schools, those with 
higher composite scores based on their activities with struggling readers, had higher probabilities 
of scoring in the top quartile on their states’ reading tests. For example, the predicted probability 
of being in the top quartile for a RF school scoring at the mean on the composite was 23 percent; 
however the probability increased to 36 percent for RF schools scoring one standard deviation 
above the mean. Similarly, non-RF Title I schools scoring at one standard deviation above the 
mean increased their probability of being in the top quartile to 82 percent, compared to 70 
percent for schools scoring at the mean. 

Only one of four composite measures was statistically significantly related to the probability that 
a school scored in the top quartile on their state reading assessment. The fact that we did not 
observe stronger links between implementation of RF-aligned activities and reading achievement 
may be a result of several factors. It may be that the relationship between reading performance 
and RF-aligned implementation is stronger than is evident in this analysis, and the measures we 
used here were insufficiently sensitive to accurately depict the true strength of the relationship. 
Alternatively, it is possible that the relationship between instruction based on SBRR and student 
reading achievement is not as strong as some would suggest. Another explanation is that it may 
be the case that students need to be exposed to more years of instruction aligned with RF before 
meaningful gains in their reading achievement are manifested. 

Discussion 

In addition to the limitations to the analysis of student reading achievement described above, 
there are also two other limitations specific to the investigation of the relationship between 
reading achievement and RF-aligned activities. First, the sample of schools included in the 
analysis is a sample of convenience, despite the fact that it is drawn from nationally 
representative samples of RF and non-RF Title I schools, because it includes only those schools 
with survey data and data from their states’ third-grade reading assessments. Therefore, we must 
caution against generalizing any observed findings to larger populations of schools. 

Second, the relational analysis uses measures that are differentially sensitive; the teacher survey 
asks detailed questions about RF activities, and the achievement measure is a very blunt 
assessment of third-grade reading performance. Further, the school sample includes only the 
extremes (highest and lowest quartiles) of the reading score distributions. We constructed a 
dichotomous (1/0) variable to indicate whether a school’s score was in the top quartile to try to 
standardize reading achievement across states as best we could, given the available data. 
However, this coarse measure may not capture schools’ improved reading performance. 

In light of these limitations, the findings (or lack thereof) about the relationship between RF- 
aligned instruction and reading achievement should be viewed with caution. 
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Summary 



There is limited evidence that students in RF schools improved their reading performance more 
quickly than their counterparts in non-RF Title I schools. A pooled analysis across states 
indicates that RF schools gained more, on average, from pre- to post-RF implementation than 
non-RF Title I schools on their states’ third-grade reading assessments, a statistically significant 
yet small difference (average effect sizes: .17-.21 ,P< .001 for all four analytic methods, 
corresponding to a 2.4 to 3.0 percentage point difference). In 12 of 24 states the improvement in 
third-grade reading performance among RF schools was statistically significantly larger than in 
non-RF Title I schools for at least one method. Similarly, for fourth-grade reading performance, 
a pooled analysis across states showed a statistically significant increase, pre- to post-RF 
implementation, in RF schools compared to non-RF Title I schools. The improvement in fourth- 
grade reading performance among RF schools was statistically significant in six of 17 states for 
at least one of the methods. 

There is little evidence of a relationship between schools’ implementation of RF-aligned 
activities and their levels of reading performance. Of all four composite measures related to 
activities aligned with Reading First strategies (classroom reading instruction, strategies to help 
struggling readers, participation in professional development and uses of assessment to inform 
instruction), only one, strategies to help struggling readers, was statistically significantly related 
to the probability that a school is in the top quartile in its state reading assessment. 
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Chapters: Conclusions 

The Reading First Program addresses the national problem of underachievement in literacy in the 
early elementary grades by promoting the use of research-based reading programs in K-3 
classrooms. The central goal of the Reading First Program is to ensure that all students read well 
by the end of grade three, thereby setting the stage for students to reach their full academic 
potential in subsequent grades. 

The Reading First Implementation Evaluation: Interim Report presented initial evidence that the 
Reading First Program was being implemented in districts and schools as intended by the 
legislation, based on surveys administered during the 2004-05 school year. Findings from the 
current analyses of 2006-07 survey data suggest that RF schools continue to implement the 
program as intended. Specifically, three major conclusions can be drawn from the findings 
described in this report. First, reading programs implemented in RF schools differ from those in 
non-RF Title I schools in several important ways. Second, from 2004-05 to 2006-07, reading 
programs in non-RF Title I schools have changed such that they increasingly aligned with the 
principles of Reading First. Third, based on pooled analyses of states’ reading assessment 
scores, there is limited evidence that third-and fourth-grade students in RF schools improved 
their reading performance over time more quickly than did their counterparts in non-RF Title I 
schools. 

The current findings suggest that there are meaningful differences in the reading programs 
implemented in RF and non-RF Title I schools. Reading First schools’ K-3 classrooms devote 
more time to reading instruction than those in non-RF Title I schools, representing almost two 
additional hours per week of reading instruction for students in RF schools. Further, RF schools 
are more likely to use reading materials and instructional practices aligned with scientifically 
based reading research than their counterparts in non-RF Title I schools. RF schools are more 
likely to have reading coaches who assist teachers in implementing their reading programs and to 
have their teachers participating in reading-related professional development. They are also 
more likely to report using assessments to guide instruction and placing struggling readers in 
intervention programs. 

While there are some notable differences between reading programs in the two types of schools, 
there is evidence of increased occurrence of RF activities in non-RF Title I schools in that these 
schools also report activities increasingly aligned with the principles of Reading First. In 2006- 
07, staff in non-RF Title I schools reported that a higher proportion of scientifically based 
teaching materials and strategies were central to their instruction than had in 2004-05. This 
increased use of materials and strategies aligned with scientifically based reading research 
(SBRR) is coupled with teachers reporting increased participation in professional development in 
the five dimensions of reading instruction (phonemic awareness, phonics, vocabulary, fluency 
and comprehension). These findings are consistent with findings reported by the Center on 
Education Policy on participation of over 3,000 non-Reading First districts in state-led Reading 
First professional Development (Center on Education Policy, 2007). 
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In addition, teachers in non-RF Title I schools reportedly have increased their activities to help 
struggling readers by greater use of diagnostic assessments to identify struggling readers and by 
increasingly placing these students in intervention programs. Further, from 2004-05 to 2006-07, 
both RF and non-RF Title I schools report significant and substantial increases in the percentages 
of teachers reporting that time is set aside to coordinate with ELL staff about reading 
interventions provided to struggling readers. Finally, an increasing percentage of teachers in 
non-RF Title I schools reported needing additional professional development in using 
assessments to guide instruction and to better assist struggling readers, which provides some 
evidence of increased attention to these issues in non-RF Title I schools. These findings provide 
evidence that while RF may have initially represented a departure from standard practice for 
teaching reading in the early grades, the principles of RF are increasingly becoming the norm in 
low achieving schools. 

Despite the increasing similarities between RF and non-RF Title I schools’ reading programs, 
there is limited evidence from this study that student reading achievement is improving more 
rapidly in RF schools than in non-RF Title I schools. However, these analyses have several 
limitations (described in Chapter 8) most notably a) the non-equivalence of the comparison 
schools to the RF schools, and b) the bluntness of the reading achievement measure (e.g., the 
proportion of students at or above a state-specific predetermined criterion). Also, the survey 
findings indicate that RF-like practices are increasingly reported by staff in non-RF Title I 
schools. This could, in effect, reduce differences in student reading performance between the 
two groups of schools. 

Further research is necessary to apply more rigorous evaluation designs and analytic techniques 
to assess the extent to which Reading First improves student reading achievement. The recently 
released Reading First Impact Study: Interim Report presented findings based on a more 
rigorous design (regression discontinuity) in a sample of over 240 schools; and that report 
indicated no significant impacts on student reading achievement for students in grades 1, 2 and 3. 

The Reading First Implementation Evaluation has provided an informative, broad overview of 
reading programs in a nationally representative sample of RF schools at two points in time, 
compared to non-RF Title I schools. However, more research is needed to understand the 
content of these programs more deeply. For example, while we learned that there is increased 
attention to the needs of struggling readers, and that staff in both RF and non-RF Title I schools 
increasingly report placing these students in interventions, little is known about the content and 
features of these interventions. The currently available evidence that these students are being 
served does not address whether their needs are ultimately being met through use of research- 
based, high quality materials and instructional strategies. Having a better understanding of the 
specific interventions used to serve struggling students would add to emerging evidence that 
some intervention programs hold promise in raising the reading achievement of struggling third- 
and fifth-graders (Torgeson, et. ah, 2006). Further, while we know that school districts play a 
key role in shaping reading programs and providing professional development, data available to 
date provide few details about the specific roles played by district staff The ongoing Reading 
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First Implementation Study: 2008-09 is designed to address these and other in-depth questions 
that arise from the findings presented in this report. 
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Appendix A: The Study Samples and the Sampling 
Weights 

The School and Respondent Samples 



The study’s evaluation questions require sampling from three distinct groups of schools: (1) new 
RF schools (schools funded in January 2004 or later), (2) mature RF schools (schools funded on 
or before Dec. 31, 2003), and (3) non-RF Title I, schoolwide project (SWP) schools — schools in 
which at least 40 percent of the students are eligible for free or reduced-price lunches. We 
limited the non-RF Title I sample to SWP schools because that is the population from which RF 
schools are typically drawn. To identify the populations of RF and non-RF Title I schools from 
which to select the study samples, we constructed sampling frames using data provided by the 
U.S. Department of Education Schools that did not have at least three of the target grades (K- 
3) were excluded from the sampling frame. 

Each of the three populations described above was sampled to yield nationally representative 
samples of schools. In order to ensure that that the three samples were representative of their 
respective populations, we stratified each group of schools into four census regions and four 
levels of school size, in which size represents the number of students in each school; this process 
created 16 strata for each sample. All RF schools under the jurisdiction of the Bureau of Indian 
Affairs (BIA) were included in our sample in an additional stratum. We sorted the schools within 
each stratum by state and urbanicity (four levels) to ensure that each sample would provide a 
systematic representation on these demographic characteristics. Finally, we selected a systematic 
sample of schools from each stratum. 

Exhibit A.l summarizes the RF and non-RF Title I samples starting with the population and 
ending with the respondents in the 2005 data collection. For Reading First, we began with a 
population of 3,91 1 schools in the fall of 2004; these schools were identified as RF schools in the 
U.S. Department of Education’s database. To construct samples of newly funded and mature RF 
schools, we sorted schools into the two groups based on the date when schools’ districts received 
their RF subgrants (There was no school-level information indicating when schools received RF 
fiinds from their districts.). 

Schools were designated as new or mature based on the Reading First program guidelines to 
states for their annual performance reports; “. . .for schools receiving grants between July 1 and 
December 3 1 of any reporting period, the current school year will be considered in the first year 
of implementation.” Therefore, schools in districts that were awarded subgrants before Dec. 31, 
2003 were designated as mature, because data collection occurred in spring 2005, when to the 



The Southwestern Educational Development Laboratory (SEDL) maintained a database for the Lf.S. Department 
of Education that tracked awards of Reading First grants and subgrants to states and districts. We obtained this 
database in October 2004 to identify the then current population of RF schools. To identify the appropriate 
population of Title I schools, we relied on the Common Core of Data (CCD) — a database that contains relevant 
demographic information on all schools nationwide. At the time we drew the sample, the CCD included data 
from the 2002-03 school year. 
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best of our knowledge those schools were in at least their second year of implementation. All 
other RF schools were classified as newly funded as of the beginning of the first data collection 
in 2005. The 1,092 responding schools consisted of 545 mature and 547 newly funded schools. 
However, after conducting the 2005 data collection, we used principals’ survey responses to 
verify and correct, if necessary, these classifications. Approximately 88 percent of these schools 
were correctly classified as new or mature. Most classification errors reflected inaccurate 
designations as mature; principals’ survey responses showed that because their schools were in 
their first year of implementation, they should be classified as “new.” The RF sample included 
642 newly funded and 450 mature schools. For non-RF Title I schools, we began with a 
population of 14,684 schools from which staff from 541 schools returned surveys in 2005 
(Exhibit A.l). 



Exhibit A.1 

The RF and Non-RF Titie I Schooi and Respondent Sampies for the Spring 2005 Data Coiiection 



RF Schoois 



2,441 



1,717 



Non-RF Titie I Schoois 




Teachers 



14,684 



551 



517 



2,027 



619 



426 



Reading 

Coaches 



273 



Source: Abt Associates’ Receipt Tracking File (2005) and Principal Survey (2005). 



The shaded boxes (642 newly- funded RF schools and 541 non-RF Title I schools) represented 
the potential sample of schools to be surveyed in 2007. The responding sample was reduced to 
579 RF and 439 non-RF Title I schools due to refusals, nonresponse, ineligibility (i.e., no longer 
a Reading First or Title I school). (See Exhibit 1.2 in Chapter 1.) 
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The Principal, Reading Coach, and Teacher Samples 



In preparation for the 2005 survey data collection, we obtained complete teacher rosters (grades 
K-3) from each school successfully recruited into the study. Each school’s principal was 
included in the evaluation. Similarly, all reading coaches, typically one per school, were included 
in the evaluation sample. Staff roster information was updated prior to the 2007 data collection. 
The teacher sample included one randomly selected teacher from each grade K-3. Some study 
schools had fewer than four teachers in the sample because the school only had three of the four 
target grades (i.e., a K-2 school), or because the school has only combined classrooms (i.e., K-1, 
1-2, 2-3). When possible, teachers completing surveys in 2005 were included as potential 
respondents again in the 2007 data collection. Approximately half of the teachers surveyed in 
2007 were respondents in 2005. 

Response Rates 

Exhibit A.2 presents response rates at the school and individual respondent levels for the 2007 
survey data collection. About 78 percent of the schools returned completed surveys from all of 
the selected respondents within their schools; another 12 percent of schools completed all but 
one of the surveys and only 41 schools, or four percent, were nonrespondents, returning no 
surveys. Response rates were somewhat higher for RE schools compared to non-RF Title I 
schools; 90 percent of RF schools were either complete or missing only one survey, compared to 
84 percent for non-RF Title I schools. It is important to note here that RF schools are required as 
part of their acceptance of a subgrant to participate in a national evaluation. In addition, non-RF 
Title I schools in districts that received RF subgrants are also required to participate in such an 
evaluation. However, participation is not required of non-RF Title I schools that are in districts 
that have not received RF subgrants. 



Exhibit A.2 

Survey Data Coiiection Response Rates for RF and Non-RF Titie i Schoois, 

2006-07 Schooi Year 

Totai RF Schoois Non-RF Titie I Schoois 



Response Rate Response Rate Response Rate 





Number 


Percent 


Number 


Percent 


Number 


Percent 


Response Status 


Complete 


826 


78% 


492 


83% 


334 


70% 


Almost complete 


130 


12 


62 


10 


68 


14 


Partial 


62 


6 


21 


5 


41 


9 


Nonrespondents 


41 


4 


10 


2 


31 


7 


Total 


1,059 


100 


585 


100 


474 


100 


Type of Respondent 


Principal 


974 


91% 


554 


94% 


420 


88% 


Teachers (K-3) 


3,793 


91 


2,190 


94 


1,603 


86 


Reading coach 


813 


95 


571 


97 


242 


89 


Total 


5,580 


91 


3,315 


95 


2,265 


87 



Source: Abt Associates’ Receipt Tracking File (2007). 
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The response rate across all types of respondents and all schools was 91 percent; of 6,108 
potential respondents, 5,580 individuals returned completed surveys. For teachers in RF schools 
response rates were approximately 94 percent across the four grade levels, compared to 86 
percent for teachers in non-RF Title I schools. 

Sampling Weights 

School-level Weights 

The construction of the school weights is based on the original recruitment sample of 1,861 
schools (1,143 RF and 718 non-RF Title I schools). Based on the stratification described above, 
schools from each stratum were weighted to represent that stratum’s population of schools and 
then adjusted for nonresponse and ineligibility. This school-level weight was constructed for the 
1,633 schools (1,092 RF and 541 non-RF Title I) that returned at least one survey in 2005. The 
same method was used to construct weights for principals and for reading coaches. Because, in 
2005, we had principal surveys from 1,574 rather than 1,633 schools, and reading coach surveys 
from 1,318 schools rather than 1,633 schools, these weights were adjusted for nonresponse at the 
principal and reading coach levels. In 2007, these three sets of weights were adjusted to account 
for refusals, ineligibility and nonresponse. 

Exhibit A.3 displays the unweighted respondent samples of schools, principals and reading 
coaches, based on the 2007 data collection. There were 1,018 schools that completed surveys in 
2007; these schools represent national populations of 1,555 RF schools and 12,909 non-RF 
Title I schools. Note that the principal respondent sample is adjusted to represent the same 
national populations of schools (since all schools have principals). Not all schools, particularly 
non-RF Title I schools, have reading coaches, this is reflected in the smaller population estimates 
for reading coaches. 



Exhibit A.3 



Unweighted and Weighted 2007 Sample Counts for RF and Non-RF Title I 



SchoolSj_Principal£^n£_Readin2^ 



Respondent 


Unweighted Sample 


Weighted Population 


School 


RF 


579 


1,555 


Non-RF Title 1 


439 


12,909 


Total 


1,018 


14,464 


Principal 


RF 


552 


1,555 


Non-RF Title 1 


413 


12,909 


Total 


965 


14,464 


Reading. Coach 


RF 


569 


1,540 


Non-RF Title 1 


236 


7,391 


Total 


805 


8,931 


Source: Abt Associates SAS Analytic datasets (2007). 
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Teacher-level Weights 



As part of the analysis of the 2005 survey data, we eonstrueted teaeher weights for two reasons. 
First, some questions on the teaeher surveys ask about classroom rather than school activities, 
and because we have a national sample of classrooms in RF and non-RF Title I schools, we 
wanted to be able to generalize to this classroom population. Second, by design, we had only one 
teacher respondent per grade per school; this teacher represents the population of teachers in that 
school at that grade. That population can vary substantially — from one teacher to 10 or 12 
teachers. If we applied the school weights to teacher responses, all teachers’ responses would 
have equal weight, regardless of the actual numbers of teachers at that grade level. For these 
reasons, we constructed a set of teacher weights that allowed us to generate estimates for the 
population of teachers in RF and non-RF Title I schools at each of the four target grades (K-3). 

The teacher weights were constructed to represent the number of teachers in their school at their 
grade levels. These weights were also adjusted to include teachers who teach multiple grades in 
their school. The “within-school” teacher weights are multiplied by the school-level weight in 
order to represent the population of teachers across all RF and non RF Title I schools. For 
example, if a first-grade teacher in our sample was in a school with three other first-grade 
teachers, her ‘within-school’ weight would be four. If the school-level weight for this school 
were five, then teacher-level weight would be 20 (five * four). The teacher weights were 
recomputed after the 2007 data collection to account for refusals, nonresponse and ineligibility. 
Exhibit A.4 presents the sums of both the teacher weights and the within-school teacher weights. 



Exhibit A.4 



Weighted Estimates of the Popuiation of Teachers in RF and Non-RF Titie i Schoois, by 
Grade 



Weighted Number of Teachers 




RF Schools 


Non-RF Title 1 Schools 






Within-school 




Within-school 


Grade Level 


Teacher Weight 


Teacher Weight 


Teacher Weight 


Teacher Weight 


Kindergarten 


5,937 


2,152 


45,207 


1,515 


1st grade 


6,328 


2,286 


47,963 


1,605 


2nd grade 


6,114 


2,205 


45,125 


1,512 


3rd grade 


5,605 


2,027 


42,988 


1,438 


Total 


23,984 


8,670 


181,283 


6,070 



Source: Abt Associates’ SAS analytic datasets (2007). 



Within-school weights were used in all teacher-level analyses comparing 2005 and 2007 
responses; as reported in Chapter 1, there was a high level of attrition (approximately 50 percent) 
in the teacher sample from 2005 to 2007. Consequently, we are unable to link individual 
teachers from year to year.’’ This makes fitting a multilevel model (where survey time point is 
nested within teachers, which are then nested within schools) that accounts for a combined 
within-school teacher/school weight infeasible (as was done in the Interim Report). Therefore, 
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Because the sample is drawn at the school level, this does not undermine our ability to generate nationally 
representative estimates of characteristics of reading programs in RF and non-RF Title I schools. 
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individual teacher responses were first weighted using within-school teacher weight. For 
example if, in a particular school there are eight first-grade teachers, the first-grade teacher 
respondent would have a within-school weight of eight. Second, school-level averages were 
computed based on the weighted sum of teacher responses within each school. Last, we fit 
statistical models to estimate comparative differences using the school-level averages and 
weighting them using school-level weights (See Appendix C). This means that, for teachers in 
non-RF Title I schools, the findings presented in the Interim Report, do not necessarily match the 
2005 findings presented in the Final Report, even though the underlying samples are equivalent. 
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Exhibit B.2.1 

School Enrollment and Urbanicity in RF and Non-RF Title I Schools, 2006-07 School Year 

RF Non-RF Title I 

Schools Schools 

Mean Standard Error Mean Standard Error p-value 

Enrollment 484 12.0 453 11.5 0.068 





Percent 


Standard Error 


Percent 


Standard Error 


p-value 


Very small (1-99) 


4% 


0.78% 


3% 


0.84% 


0.825 


Small (100-249) 


13 


1.40 


16 


1.72 


0.194 


Medium (250-499) 


41 


2.25 


44 


2.44 


0.448 


Large (500-749) 


28 


2.16 


28 


2.24 


0.926 


Very large (750+) 


14 


1.73 


9 


1.46 


0.036 




Locale 












Urban 


40 


2.35 


36 


2.47 


0.365 


Suburban 


35 


2.42 


35 


2.23 


0.900 


Rural 


25 


1.87 


29 


2.24 


0.231 



Source: Principal Survey (2007); Question A3 a. 



Exhibit B.2.2 



Mobility Rates, Attendance Rates and Changes in Enrollment in RF and Non-RF Title I 
Schools, 2006-07 School Year 





RF Schools 


Non-RF Title 1 Schools 




Change in Enrollment in 
Last Five Years 


Percent 


Standard 

Error 


Percent 


Standard 

Error 


p-value 


Decreased 


39% 


2.43% 


40% 


2.24% 


0.806 


Remained Stable 


31 


2.10 


28 


2.22 


0.322 


Increased 


29 


2.10 


32 


2.30 


0.298 


Mobility Rate 


18 


0.69 


17 


0.86 


0.612 


Attendance Rate 


93 


0.55 


93 


0.59 


0.801 



Source: Principal Survey (2007); Questions A3c, A3b, and A4 
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Exhibit B.2.3a 



Years of Experience for Staff in RF and Non-RF Title I Schools, 2006-07 School Year 



RF Schools 


Non-RF Title 1 Schools 








Standard 




Standard 






Mean 


Error 


Mean 


Error 


p-value 


Principals 












Years experience as principal 


7.2 


0.27 


8.2 


0.32 


0.019 


Years in this school 

Teachers 


4.8 


0.19 


5.7 


0.25 


0.007 


Years experience 


14.1 


0.29 


15.5 


0.34 


0.001 


Years in this school 

Reading Coaches 


9.3 


0.20 


10.9 


0.27 


0.000 


Years experience as a teacher 


17.7 


0.64 


16.8 


0.67 


0.335 


Years in this school 


9.7 


0.52 


9.3 


0.57 


0.661 


Years as reading coach in this school 


3.3 


0.40 


4.5 


0.66 


0.107 






Standard 




Standard 






Percent 


Error 


Percent 


Error 


p-value 


Schools with Reading Coaches ^ 


99% 


- 


57% 


- 


— 


Principals in this school three years or 
less 


51* 


2.30 


44 


2.40 


0.036 



^ These estimates are eomputed by dividing the weighted N of sehools with reading eoaehes by the total weighted N of sehools 
(for eaeh group of sehools, RF and non-RF Title I). 

Souree: Prineipal Survey (2007); Questions Aland A2; Teaeher Survey (2007); Question Al; Reading Coaeh Survey (2007); 
Questions Al, A2, and A3. 
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Exhibit B.2.3b 

Percent of Staff that Are in Their Current RF and Non-RF Title I Schools for Three Years or Less, 
2006-07 School Year 



Percent 




Istyear in this 2^ years in 1st year in this 2^ years in 1st year in this 2-3 years in 

school this school school this school school this school 

Principals Teachers Reading Coaches 

Source: Principal Survey (2007); Questions Aland A2; Teacher Survey (2007); Question Al; Reading Coach Survey (2007); 
Questions Al, A2, and A3. 



Exhibit B.2.4 



Preservice Training on the Five Dimensions of Reading: Teachers’ Ratings on 
Preparedness, 2006-07 School Year 







Teachers 






RF Schools 


Non-RF Title 1 Schools 




Dimension 


Mean Self- 
Rating 


Standard 

Error 


Mean Self- 
Rating 


Standard 

Error 


p-value 


Phonemic Awareness 


3.1 


0.04 


3.1 


0.04 


0.397 


Decoding 


3.2 


0.03 


3.2 


0.04 


0.571 


Vocabulary 


3.3 


0.03 


3.4 


0.03 


0.028 


Comprehension 


3.4 


0.03 


3.5 


0.03 


0.074 


Fluency 


3.0 


0.04 


3.1 


0.04 


0.020 


Source: Teacher Survey (2007); Question A3. 
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Exhibit B.2.5 



Student Characteristics in RF and Non-RF Title I Schools, 2006-07 School Year 

RF Schools Non-RF Title I Schools 

Standard Standard 





Percent 


Error 


Percent 


Error 


p-value 


Receive Special Education 
Services 


8% 


0.28% 


8% 


0.33% 


0.861 


Receive ESL Instruction 


18 


1.11 


11 


0.88 


0.000 


Instruction in Language Other 
than English 


6 


0.68 


4 


0.71 


0.095 


Reading at or above grade level 


54 


1.01 


60 


1.12 


0.000 



Source: Principal Survey (2007); Question A5b-A5e. 



Exhibit B.2.6 

Size of Reading First Grant, Per School and Per Pupil, 2004-05 School Year 



Newly Funded Reading First 


Mature Reading First 


Schools 


Schools 


Standard 


Standard 


Mean Error 


Mean Error p-value 



RF Grant Amount 

Per School $179,136 

Per Pupil (K-3) 903 


$5295 

43 


$127,175 

607 


$4507 

50 


0.000 

0.000 


Source: Principal Survey (2005); Questions A4 and B6. 
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Exhibit B.2.7 



Nonfinanciai Externai Assistance for K— 3 Reading Program Activities in RF and Non-RF Titie I 
Schoois, 2006-07 Schooi Year 





RF Schools 


Non-RF Title 1 Schools 




Type of Assistance 


Percent 


Standard 

Error 


Percent 


Standard 

Error 


p-value 


Planning professional development 


76% 


1.99% 


65% 


2.38% 


0.000 


Interpreting assessment results 


75 


1.98 


65 


2.35 


0.001 


Conducting classroom observation 


73 


2.08 


46 


2.51 


0.000 


Providing technical assistance in implementing 
core reading programs 


65 


2.19 


45 


2.50 


0.000 


Selecting professional development providers 


63 


2.29 


49 


2.54 


0.000 


Selecting assessment instruments 


54 


2.33 


54 


2.48 


0.898 


Selecting instructional programs/materials 


52 


2.33 


53 


2.50 


0.794 


Reviewing reading program effectiveness 


64 


2.23 


42 


2.50 


0.000 


Conducting demonstration lessons 


57 


2.31 


45 


2.52 


0.001 


Diagnosing needs of struggling readers 


58 


2.30 


51 


2.52 


0.027 


Setting up intervention programs for struggling 
readers 


58 


2.32 


52 


2.51 


0.111 


Providing technical assistance for using 
supplementary reading materials 


61 


2.31 


47 


2.52 


0.000 


Conducting needs assessment for professional 
development 


48 


2.33 


36 


2.40 


0.000 


Leading teacher study groups 


37 


2.20 


29 


2.33 


0.011 


Recruiting staff with reading expertise 


28 


2.15 


24 


2.16 


0.126 


Source: Principal Survey (2007); Question B2. 
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Exhibit B.2.7a 

Sources of Funding for K-3 Reading Programs in RF and Non-RF Title I Schools, 2006-07 

School Year 

RF Schools Non-RF Title I Schools 

Standard Standard 



Source of Funding 


Percent 


Error 


Percent 


Error 


p-value 


District general funds 


82% 


1.8% 


91% 


1.4% 


0.000 


State funds for reading programs 


56 


2.3 


54 


2.5 


0.461 


State textbook funds 


45 


2.3 


53 


2.5 


0.021 


Title 1 


91 


1.3 


98 


0.8 


0.000 


Title II 


40 


2.2 


47 


2.5 


0.033 


Title III 


23 


2.0 


25 


2.1 


0.371 


Comprehensive School Reform 


9 


1.4 


6 


1.2 


0.165 


21st-Century Community Learning Centers 


17 


1.7 


13 


1.7 


0.071 


Reading First 


99 


0.1 


0 


0.3 


0.000 


Professional dev. funds 


49 


2.3 


52 


0.3 


0.391 


Private grants 


9 


1.3 


17 


0.2 


0.001 



School Size (Enrollment) 


Mean N of 
Sources 


Standard 

Error 


Mean N of 
Sources 


Standard 

Error 


p-value 


1-249 students 


4.7 


0.22 


4.5 


0.17 


0.663 


250^99 


5.2 


0.14 


4.3 


0.13 


0.000 


500-749 


5.5 


0.18 


4.7 


0.16 


0.002 


750+ 


5.2 


0.26 


5.0 


0.37 


0.785 


Total 


5.2 


0.09 


4.5 


0.09 


0.000 



Source: Principal Survey (2007), Question Bl. 
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Exhibit B.3.1 



Daiiy Scheduied Reading Biock and Mean Length of the Reading Biock for RF and Non-RF Titie i 
Schoois, 2004-05 and 2006-07 Schooi Years 







2004-05 


2006-07 


2006- 


-07 - 2004-05 


















Difference 








Grades 


RF-Title 1 


Standard 




RF-Title 1 


Standard 




in 


Standard 






Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Schools with a 
reading block 


K 


11% 


1.41% 


0.000 


9% 


1.30% 


0.000 


-2.6% 


1.30% 


0.130 


1-3 


7 


1.06 


0.000 


6 


1.03 


0.000 


-1.6 


1.31 


0.229 


Schools with 
reading blocks < 


K 


-24 


2.28 


0.000 


-25 


2.34 


0.000 


-0.3 


3.10 


0.930 


90 minutes 


1-3 


-22 


1.64 


0.000 


-20 


1.65 


0.000 


2.2 


2.26 


0.340 












Mean length of 
reading block (in 


K 


8.8 


1.85 


0.000 


7.6 


1.90 


0.000 


-1.2 


2.41 


0.621 


minutes) 


1-3 


12.2 


1.76 


0.000 


13.6 


1.87 


0.000 


1.4 


2.02 


0.501 



Source: Reading Coach Survey and Principal Survey (Question D2 in 2005, Question D1 in 2007) 



Exhibit B.3.2 



Average Minutes Per Day of Reading Instruction in RF and Non-RF Titie i Schoois, 2004-05 and 
2006-07 Schooi Years 





2004-05 


2006-07 


2006- 


-07 - 2004-05 






RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


P- 

value 


Difference 

in 

Difference 


Standard 

Error 


P- 

value 


Kindergarten 


24.2 


2.14 


0.000 


24.6 


2.15 


0.000 


0.5 


2.61 


0.863 


Grades 1-3 


19.9 


1.42 


0.000 


21.4 


1.50 


0.000 


1.6 


1.82 


0.389 



Source: Teacher Survey (Question Cl in 2005, Question B1 in 2007) 



Exhibit B.3.3 

Changes to Reading Program Materiais for RF and Non-RF Titie i Schoois, 2004-05 and 2006-07 
Schooi Years 





2004-05 


2006-07 


2006- 


-07 - 2004- 


-05 


Changes to 
Reading Programs 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Adopted a new core 
reading program 


24% 


2.59% 


0.000 


-14% 


1.79% 


0.000 


-37.6% 


3.17% 


0.000 


Added a new 
intervention 
program for 
struggling readers 


31 


2.71 


0.000 


-6 


3.16 


0.124 


-37.2 


4.15 


0.000 


Added new 

supplementary 

materials 


11 


2.79 


0.000 


-16 


3.16 


0.000 


-26.8 


4.05 


0.000 


Added new 
materials for ELLs 


14 


2.81 


0.000 


-12 


2.48 


0.000 


-26.3 


3.78 


0.000 



Source: Reading Coach Survey (Question C3 in 2005, C2 in 2007) and Principal Survey (Question D7 in 2005, D5 in 2007). 
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Exhibit B.3.4 



Characteristics of Reading Materiais as Reported by Reading Coaches in RF and Non-RF Titie i 
Schoois, 2004-05 and 2006-07 Schooi Years 



Characteristics of 
Reading Materiais 


2004-05 


2006-07 


2006-07 - 2004-05 


RF -Title 1 Standard 
Difference Error p-value 


RF -Title 1 Standard 
Difference Error p-value 


Difference 

in Standard 

Difference Error p-value 


K-3 classrooms 
have ample, high 
quality instructional 
materials 


14% 3.08% 0.000 


20% 3.23% 0.000 


6.2% 4.22% 0.144 


K-3 teachers are 
experiences with 
reading intervention 
materials and 
strategies 


-9 3.65 0.010 


9 3.91 0.016 


18.3 4.83 0.000 


Reading 
intervention 
materials are 
aligned with SBRR 


9 2.82 0.002 


14 2.83 0.000 


5.2 3.96 0.191 



Source: Reading Coach Survey (Question HI in 2005, Question G1 in 2007). 



Exhibit B.3.5 



Responsibiiity for Seiection of Reading Materiais in RF and Non-RF Titie i Schoois, 2004-05 and 
2006-07 Schooi Years 





2004-05 


2006-07 


2006- 


07 - 2004- 


-05 
















Difference 








RF-Title 1 


Standard 




RF-Title 1 


Standard 




in 


Standard 






Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Selection of a specific core reading program 


State 


12% 


2.61% 


0.000 


13% 


2.78% 


0.000 


0.3% 


3.48% 


0.940 


District 


-2 


2.19 


0.065 


-1 


2.21 


0.369 


1.3 


2.66 


0.625 


Principal 

Reading 


5 


3.02 


0.006 


6 


3.28 


0.007 


1.1 


3.61 


0.756 


Coach 


9 


3.60 


0.001 


12 


3.87 


0.000 


3.0 


4.40 


0.494 


Selection of Supplemental Reading Program Materials 


State 


14 


2.10 


0.000 


15 


2.37 


0.000 


0.3 


2.81 


0.908 


District 


5 


3.01 


0.339 


11 


3.22 


0.003 


5.8 


4.09 


0.156 


Principal 


-2 


2.78 


0.334 


-4 


3.05 


0.294 


-1.5 


3.86 


0.706 


Reading 

Coach 


7 


3.44 


0.030 


11 


3.67 


0.000 


3.5 


4.73 


0.466 


Selection of Intervention Reading Program Materials for use with Struggling Readers 


State 


16 


2.07 


0.000 


19 


2.37 


0.000 


2.9 


2.70 


0.277 


District 


5 


2.97 


0.462 


6 


3.21 


0.086 


1.9 


3.74 


0.607 


Principal 


-6 


2.77 


0.016 


-5 


3.00 


0.191 


1.3 


3.80 


0.740 


Reading 

Coach 


4 


3.32 


0.158 


4 


3.54 


0.073 


-0.9 


4.46 


0.841 



Source: Principal Survey (Question D4 in 2005, D2 in 2007). 
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Exhibit B.3.6 



State and District Guidance in RF and Non-RF Title I Schools as Reported by Reading Coaches, 
2004-05 and 2006-07 School Years 





2004-05 


2006-07 


2006 


-07 - 2004-05 
















Difference 






RF-Title I 


Standard 




RF-Title I 


Standard 




in 


Standard 




Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error p-value 


The district provides direction 
concerning reading instruction 


3% 


3.33% 


0.538 


6% 


3.75% 


0.126 


2.6% 


4.58% 0.570 


The state provides direction 
concerning reading instruction 


21 


3.52 


0.000 


19 


3.33 


0.000 


-2.4 


4.44 0.594 



Source: Reading Coach Survey (Question HI in 2005, Question G1 in 2007). 



Exhibit B.3.7 



Number of Core Reading Programs Used in Grades K-3 in RF and Non-RF Title I Schools, 2006-07 
School Year 





2006-07 


Number of Core Reading Programs 


RF-Title 1 
Difference 


Standard Error 


p-value 


One core reading program for each grade served, K-3 








Same program across grades 


20% 


3.11% 


0.000 


Multiple programs across grades 


-6 


1.67 


0.000 


Multiple core reading programs for at least one grade 
served, K-3 


-14 


2.94 


0.000 



Source: Reading Coach Survey and Principal Survey, Question Cl 
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Exhibit B.3.8 



Core Reading Programs Used by RF and Non-RF Titie i Schoois, 2006-07 Schooi Year 



Publisher 


Program 


Percent of Schools 


RF 


Non-RF Title 1 


American Reading Company 


100 Book Challenge 


0.0 


0.2 


Benchmark Education Co. 


Phonetic Connections 


0.0 


0.2 


Developmental Studies 


Making Meaning 


0.0 


0.8 


Center 








Elsa Hagan 


Esperanza: Multisensory Spanish Language 
Program 


0.3 


0.0 


Sarah Hawthorne 


Read With Sarah 


0.0 


0.3 


Great Books Foundation 


Jr. Great Books 


0.0 


0.2 


Hampton Brown 


Avenues Leveled Books + 


0.3 


0.0 




Phonics and Friends 


0.2 


0.0 




Reading Basics 


0.2 


0.0 


Harcourt 


Collections 


0.9 


7.6 




Rigby Reading 


2.1 


4.2 




Signatures 


0.0 


1.0 




Trofeos (Spanish version of Trophies) 


0.2 


0.0 




Trophies 


22.5 


16.0 




Unspecified 


0.5 


0.3 


Heinemann 


Balanced Literacy (for ELL students) 


0.0 


0.3 




Fountas Pinnel units of study 


0.0 


5.3 




Guided Reading 


0.0 


0.3 




Unspecified 


0.0 


0.3 


Houghton Mifflin 


Early Success/Soar 2 Success/ Si Puedo 


0.0 


0.3 




Horizons 


1.6 


3.1 




Invitation to Literacy 


1.6 


4.6 




Lectura (Spanish Version of Reading 
California) 


5.7 


2.3 




Legacy of Literacy 


4.2 


4.8 




Nation's Choice 


10.7 


2.5 




Reading 


9.8 


10.1 




Reading In Progress 


0.0 


0.2 




State Specific Edition 


3.3 


2.3 




Unspecified 


0.3 


0.5 


Language Circle Enterprises 


Project Read 


0.0 


0.5 


McGraw-Hill 


Foro Abierto (Spanish version of Open Court) 


0.8 


0.0 




Kaleidoscope 


0.3 


0.0 




Lectura 


1.0 


0.0 




Open Court 


15.4 


9.8 




Reading 


7.0 


10.5 




Reading Mastery 


5.8 


4.1 




Spotlight on Literacy 


0.0 


2.0 




Treasures/Triumphs 


0.0 


1.7 




Unspecified 


0.3 


0.2 


McGraw-Hill/SRA 


Corrective Reading 


0.3 


0.0 


McGraw-HillA/Vright Group 


Breakthrough to Literacy 


0.3 


0.2 


Metropolitan Teaching & 


Metro/Reading Central/other reading programs 


0.0 


0.2 


Learning Company 








Mondo 


Guided Reading 


0.0 


0.2 




Mondo Book Shop 


0.2 


0.0 


National Geographic 


Windows on Literacy 


0.0 


0.2 


NCEE 


America's Choice 


0.0 


0.5 


Orton and Gillingham 


Various 


0.0 


0.3 


Owens 


The Learning Network 


0.0 


0.2 


Pearson Learning 


Words Their Way 


0.0 


0.2 


Renaissance Learning Inc. 


Accelerated Reading 


0.0 


0.3 


Saxon 


Saxon Phonics 


0.2 


6.8 








(Continues) 
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Exhibit B.3.8 



Core Reading Programs Used by RF and Non-RF Titie i Schoois, 2006-07 Schooi Year (Continued) 



Publisher 


Program 


Percent of Schools 


RF 


Non-RF Title 1 


Saxon (Continued) 


Unspecified 


0.0 


0.4 


Scholastic 


Guided Reading 


0.2 


0.8 




Literacy Place 


0.1 


2.9 




Reading Counts 


0.0 


0.2 




Storyworks 


0.0 


0.2 




Unspecified 


0.0 


0.7 


Scott Foresman 


Celebrate Reading! 


0.0 


0.5 




Early Reading Intervention 


0.2 


0.5 




Lecture (Spanish version of Reading) 


0.8 


0.2 




Leveled Readers 


0.2 


0.0 




Literacy Works 


0.3 


3.0 




Reading 


13.0 


12.2 




Reading Street 


0.3 


0.5 




State Specific Edition 


2.4 


4.9 




Unspecified 


0.2 


0.7 


Silver Burdett Ginn 


Literature Works 


0.0 


0.6 


Sophs 


Language! 


0.3 


0.3 




Read Well 


2.8 


1.7 




Unspecified 


0.3 


0.0 


Spalding Education 
International 


Spalding 


0.0 


0.3 


Success for All 


Success for All 


3.2 


4.1 


Teachers College 


Reading and Writing Project/Workshop 


0.0 


0.5 


Voyager 


Passport 


0.5 


0.5 




Universal Literacy 


4.8 


2.2 




Unspecified 


0.1 


0.3 


Waterford Institute 


Waterford 


0.0 


0.2 


Wright Group 


Early On-the-Mark Kit 


0.1 


0.0 




Guided Reading 


0.0 


0.7 




Unspecified 


0.6 


5.4 


Other 


Unspecified 


1.6 


4.9 


Core reading program developed by teachers or other school personnel 


1.5 


7.3 


Source: Reading Coach Survey and Principal Survey, Question Cl 







Exhibit B.3.9 



Reading Coach Reports of Teacher Experience with Core Reading Program in RF and Non-RF Titie i 
Schoois, 2004-05 and 2006-07 Schooi Years 





2004-05 


2006-07 


2006 


-07 - 2004- 


-05 
















Difference 








RF-Title 1 


Standard 




RF-Title 1 


Standard 




in 


Standard 






Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


The core reading program is 
aligned with SBRR 


16% 


2.60% 


0.000 


17% 


3.04% 


0.000 


1.2% 


3.68% 


0.748 


K-3 teachers are experienced 
with the core reading program 


-9 


3.41 


0.001 


19 


3.34 


0.000 


28.2 


4.49 


0.000 



Source: Reading Coach Survey (Question HI in 2005, Question G1 in 2007). 
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Exhibit B.3.10 



Suppiementai Reading Materiais in RF and Non-RF Titie i Schoois as Reported by Reading 
Coaches, 2004-05 and 2006-07 Schooi Years 







2004-05 






2006-07 




2006- 


-07 - 2004-05 




RF-Title I 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


P- 

value 


Supplemental reading 
materials are aligned with 


10% 


3.17% 


0.001 


15% 


3.23% 


0.000 


5.0% 


4.57% 


0.271 


SBRR 

K-3 teachers are experienced 
with the supplemental reading 


-13 


3.75 


0.000 


11 


3.89 


0.002 


24.3 


5.09 


0.000 


materials 





















Source: Reading Coach Survey (Question HI in 2005, Question G1 in 2007). 



Exhibit B.3.11 



Knowiedge and Motivation of Teachers in RF and Non-RF Titie i Schoois as Reported by Reading 
Coaches, 2004-05 and 2006-07 Schooi Years 





2004-05 


2006-07 


2006 


-07 - 2004-05 
















Difference 






RF-Title 1 


Standard 




RF-Title 1 


Standard 




in 


Standard 




Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error p-value 


K-3 teachers are 
knowledgeable about 
scientifically based reading 


9% 


3.67% 


0.014 


21% 


3.66% 


0.000 


1 1 .6% 


4.76% 0.015 


instruction 

K-3 teachers are motivated to 
improve reading instruction 


3 


2.87 


0.388 


8 


3.13 


0.017 


4.2 


3.96 0.290 



Source: Reading Coach Survey (Question HI in 2005, Question G1 in 2007). 
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Exhibit B.3.12 



Type and Frequency of Collaboration about Reading in RF and Non-RF Title I Schools, 2004-05 and 
2006-07 School Years 





2004-05 


2006-07 


2006- 


-07 - 2004- 


-05 


RF-Title I 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


P- 

value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Collaborate on reading lesson planning and instruction 


Not at all 


-7% 


1.45% 


0.000 


-7% 


1.39% 


0.000 


-0.1% 


1.62% 


0.974 


Monthly or less 


4 


1.69 


0.039 


7 


1.92 


0.000 


3.9 


2.21 


0.080 


Once a week or more 


7 


1.86 


0.001 


4 


2.16 


0.033 


-2.2 


2.29 


0.330 


Informally, as needed 


-3 


1.15 


0.005 


-5 


1.16 


0.000 


-1.6 


1.54 


0.307 


Observe reading instruction in other classrooms 


Not at all 


-5 


1.85 


0.001 


-10 


1.99 


0.000 


-5.0 


2.31 


0.032 


Monthly or less 


2 


1.49 


0.144 


7 


1.58 


0.000 


4.7 


1.96 


0.016 


Once a week or more 


1 


0.34 


0.050 


0 


0.47 


0.890 


-0.8 


0.58 


0.177 


Informally, as needed 


3 


1.33 


0.011 


4 


1.50 


0.005 


1.0 


1.84 


0.582 


Help with coaching or be coached about reading by other teacher 


Not at all 


-16 


1.68 


0.000 


-18 


1.79 


0.000 


-2.1 


2.19 


0.340 


Monthly or less 


6 


1.64 


0.000 


12 


1.85 


0.000 


6.2 


2.25 


0.006 


Once a week or more 


8 


1.16 


0.000 


6 


1.32 


0.000 


-2.3 


1.51 


0.127 


Informally, as needed 


1 


1.24 


0.759 


-0 


1.41 


0.757 


-1.8 


1.91 


0.358 



Source: Teacher Surveys (Question C3 in 2005, Question B3 in 2007). 
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Exhibit B.3.13 



Centraiity of SBRR-aiigned Instructionai Activities in RF and Non-RF Titie i Schoois, 2004-05 
and 2006-07 Schooi Years 



Grade Level 


Grade 


2004-05 


2006-07 


2006- 


-07 - 2004- 


-05 


RF-Title I 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference in 
Difference 


Standard 

Error 


p-value 


Phonemic 


K 


5% 


1.2% 


0.000 


5% 


1.2% 


0.000 


0% 


1.57% 


0.980 


Awareness and 






















Decoding 


1 


5 


1.5 


0.002 


4 


1.6 


0.012 


0 


2.15 


0.981 




2 


4 


2.1 


0.030 


9 


2.1 


0.000 


4 


2.76 


0.187 




3 


3 


2.0 


0.098 


10 


2.2 


0.000 


7 


2.80 


0.013 


Comprehension 


K 


5 


1.3 


0.000 


5 


1.4 


0.001 


-1 


1.68 


0.741 




1 


3 


1.5 


0.025 


2 


1.6 


0.159 


-1 


1.83 


0.546 




2 


0 


1.4 


0.831 


1 


1.6 


0.636 


-1 


1.87 


0.769 




3 


3 


1.4 


0.030 


5 


1.6 


0.001 


2 


1.92 


0.346 


Vocabulary 


K 


5 


2.0 


0.024 


5 


2.1 


0.034 


-1 


2.59 


0.705 




1 


0 


2.0 


0.982 


3 


2.1 


0.196 


2 


2.73 


0.379 




2 


-1 


1.8 


0.429 


-1 


2.0 


0.678 


-1 


2.47 


0.619 




3 


4 


1.8 


0.024 


3 


2.0 


0.130 


-1 


2.46 


0.578 


Fluency 


K 






















1 


-2 


1.9 


0.324 


4 


1.9 


0.062 


2 


2.15 


0.295 




2 


11 


2.1 


0.000 


10 


2.2 


0.000 


6 


2.63 


0.033 




3 


13 


1.4 


0.000 


14 


1.4 


0.000 


-2 


2.93 


0.604 


Overall 


K 


6 


1.2 


0.000 


9 


1.2 


0.000 


2 


1.72 


0.368 


ovji I i|JvJoiLc; 

SBRR 


1 


7 


1.3 


0.000 


7 


1.3 


0.000 


2 


1.47 


0.099 




2 


10 


1.3 


0.000 


10 


1.4 


0.000 


0 


1.75 


0.970 




3 


2 


1.1 


0.156 


-1 


1.3 


0.587 


-1 


1.88 


0.702 


Overall 


K 


-2 


1.5 


0.251 


-1 


1.7 


0.546 


-2 


1.45 


0.122 


Composite Non- 






















SBRR 


1 


-3 


1.3 


0.032 


-1 


1.5 


0.588 


2 


2.00 


0.407 




2 


1 


1.4 


0.522 


-1 


1.6 


0.620 


2 


1.81 


0.304 




3 


5 


1.2 


0.000 


5 


1.2 


0.000 


-1 


1.95 


0.562 



Source: Teacher Surveys (Question C4 in 2005, Question B4 in 2007). 
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Exhibit B.4.1 

Methods Used to identify Students for Reading Interventions in RF and Non-RF Titie I Schoois, 2004-05 
and 2006-07 Schooi Years 

2004-05 2006-07 2006-07 - 2004-05 

Difference 





RF-Title I 


Standard 




RF-Title 1 


Standard 




in 


Standard 




Method 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Progress monitoring test 
scores in reading 


5.5% 


1.57% 


0.000 


4.4% 


1.27% 


0.001 


-1.1% 


2.32% 


0.640 


Teacher recommendations 


-1.5 


1.10 


0.009 


-2.0 


1.31 


0.125 


-0.5 


1.89 


0.808 


Scores on tests that are part of 
the reading program 


-1.4 


1.67 


0.624 


1.4 


1.91 


0.490 


2.8 


2.54 


0.275 


Diagnostic test scores in 
reading 

Reading coach 
recommendation 


1.4 


1.64 


0.256 


0.5 


1.59 


0.792 


-1.0 


2.64 


0.713 


38.8 


2.56 


0.000 


33.8 


2.75 


0.000 


-5.0 


4.37 


0.251 


Screening test scores in 
reading 

Documented classroom 
observations 


6.3 


2.19 


0.003 


3.5 


2.23 


0.126 


-2.8 


3.60 


0.438 


3.3 


2.19 


0.429 


-0.4 


2.36 


0.835 


-3.7 


3.63 


0.308 


Standardized achievement 
scores in reading 


-11.6 


2.22 


0.001 


-2.3 


2.17 


0.280 


9.3 


3.03 


0.002 


Other school staff 
recommendation 


-7.0 


2.72 


0.007 


-2.4 


3.15 


0.427 


4.6 


4.64 


0.325 


Requests from parents 


-13.9 


2.73 


0.000 


-10.8 


3.11 


0.000 


3.1 


4.57 


0.501 



Source: Principal Survey (Question E2 in 2004. 



Exhibit B.4.2 



Avaiiabiiity of intervention Services in RF and Non-RF Titie i Schoois, 2004-05 and 2006-07 Schooi 
Years 







2004-05 






2006-07 




2006-07 - 2004- 


-05 




RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Availability of reading 
intervention services 


9.1% 


2.37% 


0.002 


6.7% 


2.23% 


0.003 


-2.5% 


3.77% 


0.513 


Source: Principal Survey (Question E3 in 2004-05, Question E3 in 2006-07). 
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Exhibit B.4.3 

Teachers’ Use of Supports in the Previous Month for Struggling Readers, in RF and Non-RF Title I 
Schools, 2004-05 and 2006-07 School Years 

2004-05 2006-07 2006-07 - 2004-05 



Difference 



Supports in the Previous 


RF-Title I 


Standard 




RF-Title 1 


Standard 




in 


Standard 




Month 


Difference 


Error 


p-value 


Difference 


Error 


p-vaiue 


Difference 


Error 


p-vaiue 


Diagnostic assessment to 
determine core deficits 


20.6% 


1.60% 


0.000 


16.7% 


1.73% 


0.000 


-3.9% 


2.18% 


0.074 


Placement in materials that 
supplement the core 
reading program 


13.0 


1.53 


0.000 


11.4 


1.63 


0.000 


-1.6 


1.97 


0.413 


Work with more advanced 
peer 


0.8 


1.50 


0.582 


1.6 


1.73 


0.370 


0.7 


2.06 


0.720 


Work with reading 
specialist in small group 


7.8 


1.98 


0.000 


10.6 


2.24 


0.000 


2.8 


2.32 


0.234 


Work with tutor on 
one-to-one basis 


-7.8 


1.78 


0.000 


-7.2 


1.97 


0.000 


0.5 


2.31 


0.818 


Work with reading 
specialist 

on one-to-one basis 


1.5 


1.66 


0.372 


4.7 


1.74 


0.006 


3.3 


2.08 


0.118 


Special materials for 
parents to provide practice 


-2.6 


1.67 


0.120 


1.6 


1.85 


0.371 


4.2 


2.24 


0.060 


Placement in different level 
of core reading program 


5.9 


1.75 


0.001 


2.5 


1.94 


0.175 


-3.5 


2.27 


0.128 


Placement in separate core 
reading program 


-0.2 


1.44 


0.881 


1.2 


1.74 


0.510 


1.4 


2.01 


0.500 


Source: Teacher Survey (Question C9 a, g, j, 


k, 1, m, n in 2004-05, Question B8 a, g, j, k, 1, 


m, n in 2006-07). 
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Exhibit B.4.4 

Staff Activities to Meet the Needs of Struggiing Readers, as Reported by the Principais or Reading 
Coaches in RF and Non-RF Titie i Schoois, 2005-06 and 2006—07 Schooi Years 

2004-05 2006-07 2006-07 - 2004-05 

Difference 



staff Activity 


RF-Title I 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


in 

Difference 


Standard 

Error 


p-value 


Classroom teacher provides 
additional practice opportunities 


0.9% 


0.67% 


0.163 


0.5% 


0.78% 


0.484 


-0.4% 


0.98% 


0.693 


Classroom teacher provides 
additional direct instruction 


-0.6 


1.07 


0.556 


0.0 


1.08 


0.998 


0.6 


1.46 


0.660 


Trained aides or volunteers 
work with students during class 


3.7 


1.87 


0.048 


1.5 


2.10 


0.505 


-2.2 


2.49 


0.366 


A certified reading specialist 
works directly with students 


1.8 


2.96 


0.548 


-1.9 


3.15 


0.579 


-3.6 


3.57 


0.310 


Trained aides or volunteers 
work with students before or 
after school 


-2.7 


3.03 


0.381 


-2.7 


3.29 


0.382 


-0.1% 


3.88 


0.998 


Untrained aides or volunteers 
work with students during class 


-10.3 


2.89 


0.000 


-16.6 


2.94 


0.000 


-6.3% 


3.78 


0.094 


Untrained aides or volunteers 
work with students before or 


-3.3 


2.28 


0.150 


-7.7 


2.31 


0.001 


-4.4% 


2.98 


0.141 



after school 

Source: Reading Coach Survey and Principal Survey (Question El in 2004-05, Question El in 2006-07). 



Exhibit B.4.5 


















Percent of Teachers Setting Aside Some Time to Coordinate Interventions with Speciai Education and 
Engiish Language Learner Staff in RF and Non-RF Titie 1 Schoois, 2004-05 and 2006-07 Schooi Years 






2004-05 






2006-07 




2006 


-07 - 2004-05 
















Difference 






RF-Title 1 


Standard 




RF-Title 1 


Standard 




in 


Standard 




Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error p-value 


Coordinate reading 
interventions for struggling 
readers with special 
education staff 


-2.2% 


1.71% 


0.197 


4.0% 


1.76% 


0.027 


6.2% 


2.14% 0.004 


Coordinate reading 
interventions for struggling 
readers with English 
Language Learners (ELL) 
staff 


4.5 


3.43 


0.163 


1.0 


2.48 


0.630 


-3.5 


4.17 0.396 


Source: Teacher Survey (Question C3f and g in 2004- 


05, Question B3f and g in 2006-07). 
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Exhibit B.4.6 



Teachers’ Use of Supports in the Previous Month for Engiish Language Learners to Meet the Needs of 
Struggiing Readers, in RF and Non-RF Titie I Schoois, 2004—05 and 2006-07 Schooi Years 

2004-05 2006-07 2006-07 - 2004-05 

Difference 

Supports Used in the RF-Title I Standard RF-Title I Standard in Standard 

Previous Month Difference Error p-vaiue Difference Error p-vaiue Difference Error p-vaiue 



English as a second 
language instruction 

In classroom help in 
reading from ELL teacher 

Provide reading instruction 
in home language 



1.1% 


2.73% 


0.720 


5.0 


2.82 


0.075 


3.9 


2.66 


0.150 



■2.2% 


2.93% 


0.451 


1.1 


3.29 


0.747 


3.7 


2.95 


0.209 



-3.4% 


3.74% 


0.370 


-4.0 


3.85 


0.301 


-0.1 


3.24 


0970 



Source: Teacher Survey (Composite Question CIO a-c in 2004-05, Question B9 a-c in 2006-07). 



Exhibit B.4.7 

Use of Certified Biiinguai and ESL Teachers to Provide Recommendations on Accommodations as 
Reported by RF and Non-RF Titie I Principais and Reading Coaches, 2004-05 and 2006-07 Schooi 
Years 



2004-05 2006-07 2006-07 - 2004-05 



RF-Title 1 


Standard 


RF-Title 1 


Standard 


RF-Title 1 


Standard 


Difference 


Error 


p-value Difference 


Error 


p-value Difference 


Error p-value 



A certified special education 
teacher provides 
recommendations on 
accommodations 


-5.5% 


2.47% 


0.027 


-6.3% 


2.65% 


0.016 


-0.8% 


3.32% 


0.805 


A certified bilingual/ESL teacher 
provides recommendations on 


-1.0 


3.03 


0.737 


-9.3 


3.49 


0.010 


-8.2% 


3.79 


0.030 



accommodations 

Source: Reading Coach Survey and Principal Survey (Question El in 2004-05, Question El in 2006-07). 



Exhibit B.5.1 



Percentage of Principais in RF and Non-RF Titie I Schools Reporting Receiving Assistance for K-3 
Reading Assessment Activities, 2004-05 and 2006-07 School Years 



Type of 
Assistance 


2004-05 


2006-07 


2006- 


-07 - 2004-05 




RF-Title 1 
Difference 


Standard 

Error 


P- 

value 


RF-Title 1 
Difference 


Standard 

Error 


P- 

value 


Difference in 
Difference 


Standard 

Error 


P- 

value 


Selecting 

assessment 

instruments 


23.0% 


2.87% 


0.000 


-0.3% 


3.43% 


.930 


-23.3% 


4.20% 


0.000 


Interpreting 

assessment 

results 


16.1 


2.56 


0.000 


10.3 


3.13 


.001 


-5.8 


3.91 


0.137 



Source: Principal Survey (Questions B7b and B7f in 2005, and Questions B2b and B2f in 2007). 
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Exhibit B.5.2 



Responsibiiity for Reading Assessment Activities in RF and Non-RF Titie i Schoois, as Reported by 
Principais, 2004-05 and 2006-07 Schooi Years 





2004-05 


2006-07 


2006-07 - 2004- 


05 


RF-Title 1 
Difference 


Standard 

Error 


P- 

value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference in 
Difference 


Standard 

Error 


P- 

value 


Selecting assessment instruments 


State 


30.8% 


2.93% 


0.000 


26.3% 


3.17% 


0.000 


-4.7% 


4.02% 


0.239 


District 


-12.0 


2.51 


0.000 


-5.4 


2.77 


0.050 


6.5 


3.44 


0.061 


Principal 


-8.5 


3.11 


0.006 


-6.1 


3.41 


0.075 


2.1 


4.08 


0.603 


School’s reading 


11.7 


3.00 


0.000 


15.9 


3.28 


0.000 


4.2 


3.95 


0.290 


coach 




















Interpreting assessment results 


State 


13.8 


2.41 


0.000 


18.8 


2.61 


0.000 


4.9 


3.23 


0.128 


District 


-4.8 


3.04 


0.114 


2.1 


3.30 


0.517 


6.8 


4.03 


0.094 


Principal 


-3.5 


1.78 


0.049 


0.0 


1.52 


0.998 


3.4 


2.10 


0.103 


School’s reading 


43.7 


2.37 


0.000 


39.0 


2.46 


0.000 


-4.7 


2.94 


0.108 


coach 





















Source: Principal Survey (Questions D4d-D5d in 2005, Questions D2d-D3d in 2007). 



Exhibit B.5.3 



Time Set Aside for K-3 Teachers to Use Assessment Data to Pian Instruction in RF and Non-RF Titie i 
Schoois, 2004-05 and 2006-07 Schooi Years 





2004-05 


2006-07 


2006-07 - 2004- 


05 


RF-Title 1 
Difference 


Standard 

Error 


P- 

value 


RF-Title 1 
Difference 


Standard 

Error 


P- 

value 


Difference in 
Difference 


Standard 

Error 


P- 

value 


Once a week or 
more 


4.4% 


1.61% 


0.007 


6.1% 


1.80% 


0.001 


1.7% 


2.23% 


0.449 


Once a month 


5.6 


1.29 


0.000 


7.8 


1.55 


0.000 


2.2 


1.89 


0.241 


5-8 times 


-0.8 


0.90 


0.401 


3.7 


1.07 


0.001 


4.4 


1.37 


0.001 


1-4 times 


-2.9 


1.52 


0.057 


-7.6 


1.62 


0.000 


-4.8 


2.11 


0.024 


Any regular time 


6.3 


1.50 


0.000 


10.0 


1.38 


0.000 


3.5 


1.88 


0.062 


Not at all 


-3.5 


1.09 


0.002 


-6.7 


0.89 


0.000 


-3.1 


1.25 


0.012 


Informally, only as 
needed 


-2.8 


1.09 


0.009 


-3.0 


1.09 


0.006 


-0.1 


1.49 


0.957 


No regular time 


-6.3 


1.50 


0.000 


-10.0 


1.38 


0.000 


-3.5 


1.88 


0.062 



Source: Teacher Survey (Question C3c in 2005, Question B3c in 2007). 
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Exhibit B.5.4a 

Types of Assessments Used Most Often by Kindergarten Teachers in RF and Non-RF Title I 
Schools, by Assessment Purpose, 2006-07 School Year 

Kindergarten Teachers ^ ^ 

RF Non-RF Title I 



Assessment Purpose/ Standard Standard 



Type of Assessment 


Percent 


Error 


Percent 


Error 


p-value 


Placing or grouping of students 


Formal assessments 












Core, supplementary, or intervention reading 
program assessment 


9.2% 


1.30% 


12.4% 


1.74% 


0.136 


District-specific assessment 


3.1 


0.91 


2.7 


0.84 


0.702 


State-specific assessment 


2.0 


0.79 


2.4 


0.79 


0.717 


Other standardized assessment 


64.2 


2.03 


37.9 


2.52 


0.000 


Informal assessments 












Structured informal assessments 


1.2 


0.44 


5.4 


1.18 


0.001 


Unstructured informal assessments 


2.9 


0.85 


7.5 


1.39 


0.004 


Assessment is not used for this purpose 


1.9 


0.62 


11.5 


1.63 


0.000 


Determining student mastery of skills 


Formal assessments 












Core, supplementary, or intervention reading 
program assessment 


21.2 


1.83 


21.1 


2.13 


0.973 


District-specific assessment 


5.2 


1.07 


5.0 


1.09 


0.914 


State-specific assessment 


2.4 


0.80 


4.1 


1.02 


0.181 


Other standardized assessment 


44.1 


2.29 


24.9 


2.26 


0.000 


Informal assessments 












Structured informal assessments 


2.2 


0.78 


6.8 


1.33 


0.003 


Unstructured informal assessments 


7.2 


1.27 


18.9 


2.07 


0.000 


Assessment is not used for this purpose 


0.9 


0.48 


2.3 


0.75 


0.116 


Identifying the core deficits of struggling students 


Formal assessments 












Core, supplementary, or intervention reading 
program assessment 


11.5 


1.54 


9.8 


1.61 


0.453 


District-specific assessment 


3.3 


0.96 


2.9 


0.91 


0.781 


State-specific assessment 


2.0 


0.84 


2.6 


0.86 


0.589 


Other standardized assessment 


55.6 


2.39 


35.8 


2.55 


0.000 


Informal assessments 












Structured informal assessments 


1.7 


0.70 


3.8 


1.04 


0.101 


Unstructured informal assessments 


5.4 


1.15 


10.7 


1.66 


0.009 


Assessment is not used for this purpose 


4.2 


0.96 


17.5 


2.05 


0.000 



Percentages by grade or assessment purpose will not add up to 100 percent due to nonresponse, responses that we were not 
able to categorize, and multiple responses. 



We were not able to categorize about 10 percent of teachers’ responses. “Not able to categorize” includes responses that were 
too vague to be coded (e.g., assessment, test, eight-weeks, benchmarks, decoding, rubric, pretest, quarterly test, fluency) and 
responses that were not discemable (e.g., ELLA stands for Early Learning Literacy in Arkansas or the English Language and 
Literacy Assessment). In addition, about 20 percent of teachers’ responses were excluded from analyses because the teacher 
named more than one assessment that we were able to categorize and, thus, we did now know which to select as one assessment 
used most often. 

Source: Teacher Survey (Questions B6aa, B6ab, and B6ac in 2007). 
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Exhibit B.5.4b 












Types of Assessments Used Most Often by First-Grade Teachers in RF and Non-RF Titie 1 


Schoois, by Assessment Purpose, 2006- 


-07 School Year 












First-Grade Teachers ^ ^ 






Assessment Purpose/ 


RF 


Non-RF Title 1 






Standard 




Standard 




Type of Assessment 


Percent 


Error 


Percent 


Error 


p-value 


Piacing or grouping of students 


Formal assessments 












Core, supplementary, or intervention 
reading program assessment 


1 1 .2% 


1.52% 


16.6% 


1.92% 


0.026 


District-specific assessment 


3.2 


0.78 


1.1 


0.57 


0.032 


State-specific assessment 


2.9 


0.92 


2.4 


0.78 


0.682 


Other standardized assessment 


62.4 


2.13 


44.0 


2.67 


0.000 


Informal assessments 












Structured informal assessments 


4.6 


0.94 


9.5 


1.52 


0.005 


Unstructured informal assessments 


1.2 


0.44 


5.4 


1.16 


0.001 


Assessment is not used for this purpose 


0.8 


0.37 


4.1 


1.02 


0.002 


Determining student mastery of skiiis 


Formal assessments 












Core, supplementary, or intervention 
reading program assessment 


31.0 


2.10 


31.1 


2.41 


0.978 


District-specific assessment 


3.8 


0.83 


4.3 


1.05 


0.718 


State-specific assessment 


1.4 


0.67 


2.4 


0.78 


0.360 


Other standardized assessment 


38.9 


2.19 


21.3 


2.08 


0.000 


Informal assessments 












Structured informal assessments 


1.6 


0.51 


6.5 


1.27 


0.000 


Unstructured informal assessments 


1.9 


0.60 


6.8 


1.29 


0.001 


Assessment is not used for this purpose 


0.7 


0.42 


4.5 


1.09 


0.001 


identifying the core deficits of struggiing students 


Formal assessments 












Core, supplementary, or intervention 
reading program assessment 


14.0 


1.62 


10.1 


1.58 


0.089 


District-specific assessment 


2.7 


0.77 


1.1 


0.54 


0.083 


State-specific assessment 


2.0 


0.83 


1.9 


0.71 


0.945 


Other standardized assessment 


54.7 


2.33 


36.3 


2.48 


0.000 


Informal assessments 












Structured informal assessments 


2.6 


0.66 


11.2 


1.67 


0.000 


Unstructured informal assessments 


2.7 


0.81 


5.7 


1.21 


0.044 


Assessment is not used for this purpose 


3.4 


0.90 


13.6 


1.82 


0.000 


^ Percentages by grade or assessment purpose will not add up to 100 percent due to nonresponse, responses that we were not 


able to categorize, and multiple responses. 












We were not able to categorize about 10 percent of teachers’ responses. “Not able to categorize” 


includes responses that 


were too vague to be coded (e.g., assessment, test, eight-weeks, benchmarks, decoding, rubric, pretest, quarterly test, fluency) 
and responses that were not discemable (e.g., ELLA stands for Early Learning Literacy in Arkansas or the English Language 
and Literacy Assessment). In addition, about 20 percent of teachers’ responses were excluded from analyses because the 
teacher named more than one assessment that we were able to categorize and, thus, we did now know which to select as one 


assessment used most often. 












Source: Teacher Survey (Questions B6aa, B6ab, and B6ac in 2007). 
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Exhibit B.5.4C 



Types of Assessments Used Most Often by Second-Grade Teachers in RF and Non-RF Titie i Schoois, by 
Assessment Purpose, 2006-07 Schooi Year 

Second-Grade Teachers ^ ^ 

RF Non-RF Titie i 

Assessment Purpose/ standard Standard 

Type of Assessment Percent Error Percent Error p-vaiue 

Piacing or grouping of students 

Formal assessments 



Core, supplementary, or intervention reading 



program assessment 


9.2% 


1.39% 


17.4% 


1.93% 


0.001 


District-specific assessment 


1.5 


0.56 


1.3 


0.61 


0.896 


State-specific assessment 


0.7 


0.52 


1.6 


0.66 


0.283 


Other standardized assessment 


71.2 


2.05 


50.3 


2.55 


0.000 


Informal assessments 












Structured informal assessments 


2.6 


0.71 


7.3 


1.37 


0.002 


Unstructured informal assessments 


0.4 


0.37 


1.6 


0.68 


0.098 


Assessment is not used for this purpose 


1.0 


0.41 


4.7 


1.09 


0.001 


Determining student mastery of skiiis 


Formal assessments 












Core, supplementary, or intervention reading 
program assessment 


36.4 


2.27 


37.6 


2.56 


0.741 


District-specific assessment 


5.0 


0.88 


4.0 


1.04 


0.480 


State-specific assessment 


1.2 


0.60 


1.4 


0.61 


0.822 


Other standardized assessment 


31.1 


0.22 


21.7 


2.15 


0.002 


Informal assessments 












Structured informal assessments 


0.9 


0.38 


5.1 


1.17 


0.001 


Unstructured informal assessments 


4.0 


0.94 


7.1 


1.35 


0.057 


Assessment is not used for this purpose 


1.2 


0.62 


5.7 


1.27 


0.001 


identifying the core deficits of struggiing students 


Formal assessments 












Core, supplementary, or intervention reading 
program assessment 


14.9 


1.70 


13.2 


1.88 


0.505 


District-specific assessment 


1.6 


0.62 


1.8 


0.72 


0.882 


State-specific assessment 


0.9 


0.50 


1.9 


0.72 


0.248 


Other standardized assessment 


49.9 


2.33 


34.9 


2.58 


0.000 


Informal assessments 












Structured informal assessments 


2.9 


0.78 


9.5 


1.65 


0.000 


Unstructured informal assessments 


3.8 


1.02 


2.5 


0.85 


0.322 


Assessment is not used for this purpose 


6.3 


1.30 


16.7 


2.07 


0.000 



^ Percentages by grade or assessment purpose will not add up to 100 percent due to nonresponse, responses that we were not 



able to categorize, and multiple responses. 

We were not able to categorize about 10 percent of teachers’ responses. “Not able to categorize” includes responses that 
were too vague to be coded (e.g., assessment, test, eight-weeks, benchmarks, decoding, rubric, pretest, quarterly test, fluency) 
and responses that were not discemable (e.g., ELLA stands for Early Learning Literacy in Arkansas or the English Language 
and Literacy Assessment). In addition, about 20 percent of teachers’ responses were excluded from analyses because the 
teacher named more than one assessment that we were able to categorize and, thus, we did now know which to select as one 
assessment used most often. 

Source: Teacher Survey (Questions B6aa, B6ab, and B6ac in 2007). 
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Exhibit B.5.4d 



Types of Assessments Used Most Often by Third-Grade Teachers in RF and Non-RF Titie i 
Schoois, by Assessment Purpose, 2006-07 Schooi Year 







Third-Grade Teachers ^ ^ 










RF 


Non-RF Title 1 




Assessment Purpose/ 
Type of Assessment 


Percent 


Standard 

Error 


Percent 


Standard 

Error 


p-value 


Placing or grouping of students 


Formal assessments 












Core, supplementary, or intervention 
reading program assessment 


12.5% 


1.59% 


13.4% 


1.82% 


0.705 


District-specific assessment 


1.9 


0.70 


2.0 


0.74 


0.865 


State-specific assessment 


2.8 


0.87 


4.3 


1.08 


0.267 


Other standardized assessment 


63.5 


2.22 


47.7 


2.66 


0.000 


Informal assessments 












Structured informal assessments 


3.4 


0.86 


5.2 


1.21 


0.230 


Unstructured informal assessments 


1.4 


0.65 


1.31 


0.58 


0.964 


Assessment is not used for this purpose 


1.9 


0.72 


9.0 


1.49 


0.000 


Determining student mastery of skiiis 


Formal assessments 












Core, supplementary, or intervention 
reading program assessment 


35.2 


2.24 


28.4 


2.42 


0.039 


District-specific assessment 


6.6 


1.26 


4.1 


1.05 


0.118 


State-specific assessment 


3.7 


1.01 


7.6 


1.44 


0.027 


Other standardized assessment 


26.0 


2.02 


21.1 


2.24 


0.106 


Informal assessments 












Structured informal assessments 


1.1 


0.43 


2.2 


0.85 


0.218 


Unstructured informal assessments 


3.6 


0.83 


6.5 


1.29 


0.058 


Assessment is not used for this purpose 


2.6 


0.82 


5.9 


1.28 


0.031 


identifying the core deficits of struggiing students 


Formal assessments 












Core, supplementary, or intervention 
reading program assessment 


14.4 


1.61 


12.0 


1.75 


0.302 


District-specific assessment 


2.9 


0.95 


2.5 


0.86 


0.763 


State-specific assessment 


2.2 


0.88 


4.2 


1.11 


0.157 


Other standardized assessment 


43.1 


2.35 


28.6 


2.52 


0.000 


Informal assessments 












Structured informal assessments 


3.4 


0.91 


5.6 


1.29 


0.166 


Unstructured informal assessments 


3.2 


0.92 


5.4 


1.24 


0.162 


Assessment is not used for this purpose 


7.6 


1.34 


22.4 


2.31 


0.000 


^ Percentages by grade or assessment purpose will not add up to 100 percent due to nonresponse, responses that we were not 
able to categorize, and multiple responses. 


We were not able to categorize about 10 percent of teachers’ responses. “Not able to categorize” includes responses that 
were too vague to be coded (e.g., assessment, test, eight-weeks, benchmarks, decoding, rubric, pretest, quarterly test, fluency) 
and responses that were not discemable (e.g., ELLA stands for Early Learning Literacy in Arkansas or the English Language 
and Literacy Assessment). In addition, about 20 percent of teachers’ responses were excluded from analyses because the 
teacher named more than one assessment that we were able to categorize and, thus, we did now know which to select as one 
assessment used most often. 

Source: Teacher Survey (Questions B6aa, B6ab, and B6ac in 2007). 
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Exhibit B.5.5a 



Types of Specific Assessments Reported Used Most Often by Kindergarten Teachers in RF and 
Non-RF Title I Schools, by Assessment Purpose, 2006-07 School Year 

ab 

Kindergarten Teachers 



Assessment Purpose/ 
Type of Assessment 



RF Non-RF Title I 

Standard Standard 

Percent Error Percent Error p-value 



Placing or grouping of students 



Formal assessments 



DIBELS 


50.4% 


2.43% 


19.2% 


2.17% 


0.000 


DRA 


1.8 


0.73 


6.5 


1.35 


0.002 


TPRI 


11.2 


1.56 


4.9 


1.13 


0.001 


Determining student mastery of skills 


Formal assessments 












DIBELS 


31.2 


2.34 


11.6 


1.80 


0.000 


DRA 


0.0 


0.00 


2.9 


0.92 


0.002 


TPRI 


6.9 


1.34 


3.6 


0.98 


0.046 



Identifying the core deficits of struggling students 



Formal assessments 



DIBELS 


38.1 


2.41 


18.8 


2.15 


0.000 


DRA 


0.4 


0.43 


1.5 


0.65 


0.186 


TPRI 


8.3 


1.52 


6.1 


1.27 


0.282 



Percentages by grade or assessment purpose will not add up to 100 percent due to nonresponse, responses that we were not 
able to categorize, and multiple responses. 

We were not able to categorize about 10 percent of teachers’ responses. “Not able to categorize” includes responses that 
were too vague to be coded (e.g., assessment, test, eight-weeks, benchmarks, decoding, rubric, pretest, quarterly test, fluency) 
and responses that were not discemable (e.g., ELLA stands for Early Learning Literacy in Arkansas or the English Language 
and Literacy Assessment). In addition, about 20 percent of teachers’ responses were excluded from analyses because the 
teacher named more than one assessment that we were able to categorize and, thus, we did now know which to select as the 
one assessment used most often. 

Source: Teacher Survey (Questions B6aa, B6ab, and B6ac in 2007). 
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Exhibit B.5.5b 

Types of Specific Formal Assessments Reported as Used Most Often by First-Grade Teachers in 
RF and Non-RF Title I Schools, by Assessment Purpose, 2006-07 School Year 

First-Grade Teachers ^ ^ 

RF Non-RF Title I 

Assessment Purpose/ standard Standard 

Type of Assessment Percent Error Percent Error p-value 

Placing or grouping of students 

Formal assessments 
DIBELS 
DRA 

TPRI 

Determining student mastery of skills 

Formal assessments 
DIBELS 
DRA 

TPRI 

Identifying the core deficits of struggling students 

Formal assessments 



DIBELS 


37.3 


2.33 


20.7 


2.35 


0.000 


DRA 


0.4 


0.27 


3.2 


1.05 


0.010 


TPRI 


8.3 


1.41 


5.2 


1.24 


0.105 



24.3 


2.08 


8.6 


1.72 


0.000 


0.8 


0.52 


4.4 


1.30 


0.011 


8.1 


1.41 


2.5 


0.91 


0.001 



43.9% 


2.20% 


19.3% 


2.31% 


0.000 


3.8 


1.03 


17.3 


2.21 


0.000 


11.2 


1.58 


4.6 


1.19 


0.001 



Percentages by grade or assessment purpose will not add up to 100 percent due to nonresponse, responses that we were not 
able to categorize, and multiple responses. 

We were not able to categorize about 10 percent of teachers’ responses. “Not able to categorize” includes responses that were 
too vague to be coded (e.g., assessment, test, eight-weeks, benchmarks, decoding, rubric, pretest, quarterly test, fluency) and 
responses that were not discemable (e.g., ELLA stands for Early Learning Literacy in Arkansas or the English Language and 
Literacy Assessment). In addition, about 20 percent of teachers’ responses were excluded from analyses because the teacher 
named more than one assessment that we were able to categorize and, thus, we did now know which to select as one assessment 
used most often. 

Source: Teacher Survey (Questions B6aa, B6ab, and B6ac in 2007). 
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Exhibit B.5.5C 



Types of Specific Formai Assessments Reported as Used Most Often by Second-Grade 
Teachers in RF and Non-RF Titie i Schoois, by Assessment Purpose, 2006-07 Schooi Year 

Second-Grade Teachers ® 

RF Non-RF Title I 

Assessment Purpose/ Standard Standard 



Type of Assessment 


Percent 


Error 


Percent 


Error 


p-value 


Placing or grouping of students 


Formal assessments 












DIBELS 


43.8% 


2.29% 


15.0% 


1.96% 


0.000 


DRA 


5.8 


1.25 


17.4 


2.09 


0.000 


TPRI 


12.3 


1.60 


5.0 


1.16 


0.000 


Determining student mastery of skills 


Formal assessments 












DIBELS 


11.1 


1.52 


4.8 


1.17 


0.001 


DRA 


1.4 


0.50 


2.4 


0.85 


0.285 


TPRI 


7.1 


1.38 


2.3 


0.80 


0.003 


Identifying the core deficits of struggling students 


Formal assessments 












DIBELS 


23.3 


2.07 


12.7 


1.74 


0.000 


DRA 


1.3 


0.60 


4.5 


1.14 


0.012 


TPRI 


11.1 


1.61 


4.8 


1.12 


0.002 



Percentages by grade or assessment purpose will not add up to 100 percent due to nonresponse, responses that we were not 
able to categorize, and multiple responses. 

We were not able to categorize about 10 percent of teachers’ responses. “Not able to categorize” includes responses that 
were too vague to be coded (e.g., assessment, test, eight-weeks, benchmarks, decoding, rubric, pretest, quarterly test, fluency) 
and responses that were not discemable (e.g., ELLA stands for Early Learning Literacy in Arkansas or the English Language 
and Literacy Assessment). In addition, about 20 percent of teachers’ responses were excluded from analyses because the 
teacher named more than one assessment that we were able to categorize and, thus, we did now know which to select as one 
assessment used most often. 

Source: Teacher Survey (Questions B6aa, B6ab, and B6ac in 2007). 
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Exhibit B.5.5d 



Types of Specific Formai Assessments Reported as Used Most Often by Third-Grade Teachers in 
RF and Non-RF Titie 1 Schoois, by Assessment Purpose, 2006-07 Schooi Year 


Assessment Purpose/ 
Type of Assessment 




3 b 

Third-Grade Teachers 
RF Non-RF Title 1 


- 


Percent 


Standard 

Error 


Percent 


Standard 

Error 


p-value 


Placing or grouping of students 


Formal assessments 












DIBELS 


42.1% 


2.29% 


8.1% 


1.55% 


0.000 


DRA 


3.6 


0.98 


12.8 


1.86 


0.000 


TPRI 


9.9 


1.51 


0.3 


0.32 


0.000 


Determining student mastery of skills 


Formal assessments 












DIBELS 


9.4 


1.39 


2.5 


0.86 


0.000 


DRA 


0.7 


0.37 


5.4 


1.29 


0.001 


TPRI 


3.5 


0.97 


0.0 


0.00 


0.000 


Identifying the core deficits of struggling students 


Formal assessments 












DIBELS 


20.4 


1.98 


7.0 


1.43 


0.000 


DRA 


1.8 


0.68 


5.9 


1.30 


0.006 


TPRI 


5.3 


1.06 


0.3 


0.28 


0.000 



^ Percentages by grade or assessment purpose will not add up to 100 percent due to nonresponse, responses that we were not 
able to categorize, and multiple responses. 

We were not able to categorize about 10 percent of teachers’ responses. “Not able to categorize” includes responses that were 
too vague to be coded (e.g., assessment, test, eight-weeks, benchmarks, decoding, rubric, pretest, quarterly test, fluency) and 
responses that were not discemable (e.g., ELLA stands for Early Learning Literacy in Arkansas or the English Language and 
Literacy Assessment). In addition, about 20 percent of teachers’ responses were excluded from analyses because the teacher 
named more than one assessment that we were able to categorize and, thus, we did now know which to select as one assessment 
used most often. 

Source: Teacher Survey (Questions B6aa, B6ab, and B6ac in 2007). 
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Exhibit B.5.6 

Number of Times Most Frequently Used Assessment Is Administered Per Year as Reported by 
Teachers in RF and Non-RF Title I Schools, by Assessment Purpose, 2006-07 School Year 

Teachers 

RF Schools Non-RF Title I Schools 

Standard Standard 

Frequency Percent Error Percent Error p-value 



Placing or grouping students 



1 or 2 times 


8.7% 


0.83% 


24.7% 


1.42% 


0.000 


3 or 4 times 


53.6 


1.45 


53.7 


1.67 


0.952 


5 or more times 


37.7 


1.38 


21.6 


1.37 


0.000 


Determining student mastery of skills 


1 or 2 times 


7.9 


0.73 


11.7 


1.05 


0.003 


3 or 4 times 


32.3 


1.35 


36.9 


1.53 


0.024 


5 or more times 


59.8 


1.38 


51.4 


1.61 


0.000 


Identifying the core deficits of struggling 
students 


1 or 2 times 


11.1 


0.95 


17.1 


1.35 


0.000 


3 or 4 times 


36.1 


1.41 


44.3 


1.81 


0.000 


5 or more times 


52.7 


1.54 


38.5 


1.72 


0.000 



Source: Teacher Survey (Questions B6ba, B6bb, and B6bc in 2007). 



Exhibit B.5.7 

Length of Administration for Most Frequently Used Assessment, as Reported by Teachers in RF 
and Non-RF Title I Schools by Assessment Purpose, 2006-07 School Year 

Teachers 



Non-RF Title I 

RF Schools Schools 



Length of Administration 


Percent 


Standard 

Error 


Percent 


Standard 

Error 


p-value 


Placing or grouping students 


1-15 minutes 


63.2% 


1.48% 


41.8% 


1.61% 


0.000 


16-30 minutes 


19.7 


1.12 


30.8 


1.45 


0.000 


More than 30 minutes 


17.1 


1.12 


27.3 


1.44 


0.000 


Determining student mastery of skills 


1-15 minutes 


41.0 


1.43 


33.1 


1.53 


0.000 


16-30 minutes 


20.9 


1.10 


25.9 


1.35 


0.004 


More than 30 minutes 


38.1 


1.39 


41.0 


1.57 


0.161 


Identifying the core deficits of struggling students 


1-15 minutes 


59.0 


1.58 


40.9 


1.77 


0.000 


16-30 minutes 


19.0 


1.18 


27.9 


1.58 


0.000 


More than 30 minutes 


22.0 


1.29 


31.3 


1.68 


0.000 



Source: Teacher Survey (Questions B6da, B6db, and B6dc in 2007). 
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Exhibit B.5.8 

Accountability Purposes for Most Frequently Used Assessment, as Reported by Teachers in RF 

and Non-RF Title I Schools, by Assessment Purpose, 2006-07 School Year 

Teachers 



RF Schools Non-RF Title I 

Schools 



Accountability Purpose 


Percent 


Standard 

Error 


Percent 


Standard 

Error 


p-value 


Reading First 


95.5% 


5.85% 


18.6% 


1.31% 


0.000 


No Child Left Behind 


37.3 


1.44 


47.1 


1.55 


0.000 


Other program 


40.1 


1.43 


62.3 


1.55 


0.000 


Source: Teacher Survey (Questions B6ea, B6eb, and B6ec in 2007). 



Exhibit B.5.9 



Teachers’ Use of Assessments in RF Schools and Non-RF Title I Schools, 2004-05 and 2006-07 School 
Years 







2004-05 






2006-07 




2006-07 - 2004-1 


05 
















Difference 






Use of Assessments 


RF-Title 1 


Standard 


P- 


RF-Title 1 


Standard 


P- 


in 


Standard 


P- 


Difference 


Error 


value 


Difference 


Error 


value 


Difference 


Error 


value 


Use test results to organize 
instructional groups 


15.2% 


1.43% 


0.000 


13.0% 


1.55% 


0.000 


-2.2% 


1.73% 


0.206 


Use informal reading 
inventories 


-0.1 


1.61 


0.947 


1.8 


1.90 


0.354 


1.8 


2.26 


0.430 


Use tests to determine 
progress on skills 


7.8 


1.32 


0.000 


7.9 


1.45 


0.000 


0.1 


1.76 


0.968 


Use tests to determine who 
can benefit from the core 
reading series 


12.4 


1.67 


0.000 


16.0 


1.93 


0.000 


3.5 


2.22 


0.116 


Use diagnostic tests to 
identify students who need 
reading intervention services 


17.1 


1.52 


0.000 


14.1 


1.67 


0.000 


-3.0 


1.98 


0.129 


Use screening tests to 
identify students who need a 
supplementary reading 
program 


11.8 


1.69 


0.000 


11.0 


1.91 


0.000 


-1.0 


2.17 


0.643 


Conduct miscue analysis, 
analyzing errors students 
make while reading aloud 


1.3 


1.66 


0.433 


5.0 


1.87 


0.008 


3.6 


2.17 


0.100 



Source: Teacher Survey (Questions C6u-C6aa in 2005, Questions B5u-B5aa in 2007). 
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Exhibit B.6.1 



Responsibilities of the Reading Coach in RF and Non-RF Title I Schools, 2004-05 and 2006-07 School 
Years 



2004-05 2006-07 2006-07 - 2004-05 



Responsibilities of 
Reading Coach 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Average number of 
schools with which 
reading coach works 


-0.3 


0.10 


0.009 


-0.1 


0.09 


0.30 


0.2 


0.11 


0.162 


... 1 school 


6.9% 


2.52% 


0.006 


3.7% 


2.35% 


0.12 


-3.2% 


2.77% 


0.250 


... 2 schools 


-2.6 


2.05 


0.208 


-1.5 


1.85 


0.44 


1.1 


2.44 


0.645 


... 3 schools 


-0.1 


0.83 


0.892 


-0.2 


0.97 


0.86 


-0.1 


1.02 


0.945 


... 4+ schools 


-4.2 


1.70 


0.014 


-2.1 


1.22 


0.09 


2.1 


2.00 


0.289 


Average number of 
teachers with whom 
reading coach works 


-3.8 


0.99 


0.000 


-2.9 


1.08 


0.01 


0.8 


1.14 


0.465 


... 1-10 teachers 


1.9% 


2.68% 


0.486 


1.0% 


3.17% 


0.72 


-0.9% 


3.61% 


0.812 


... 11-20 teachers 


11.3 


3.62 


0.002 


13.5 


4.16 


0.00 


2.3 


4.85 


0.639 


... 21-30 teachers 


-4.7 


3.49 


0.179 


-6.2 


4.17 


0.15 


-1.5 


4.77 


0.756 


... 31+ teachers 


-8.4 


2.97 


0.005 


-8.4 


3.26 


0.01 


0.1 


3.51 


0.985 


Average percentage of 
time spent as reading 
coach in this school 


28.5 


2.05 


0.000 


35.9 


2.64 


0.00 


7.4 


2.69 


0.006 


... 100% time 


48.2% 


3.28% 


0.000 


56.4% 


3.59% 


0.00 


8.2% 


4.26% 


0.055 


... 75-99% time 


-1.7 


2.90 


0.568 


-6.7 


3.41 


0.05 


-5.0 


4.11 


0.224 


... 50-74% time 


-25.4 


3.51 


0.000 


-28.4 


3.99 


0.00 


-2.9 


4.64 


0.529 


... 25-49% time 


-12.8 


2.73 


0.000 


-8.7 


2.79 


0.00 


4.0 


3.42 


0.241 


... 1-24% time 


-8.4 


2.05 


0.000 


-12.7 


2.94 


0.00 


-4.3 


3.21 


0.181 



Source: Reading Coach Survey (2005); Questions Bl, B2, and B3; Reading Coach Survey (2007); Questions Bl, B2, and B3. 
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Exhibit B.6.2 



Percentage of Reading Coaches Rating Various Teacher Support Activities as Absolutely Central to 
Their Work in RF and Non-RF Title I Schools, 2004-05 and 2006-07 School Years 







2004-05 






2006-07 




2006 


-07 - 2004- 


-05 


Activity 


RF-Non- 
RF Title 1 
Difference 


Standard 

Error 


p-value 


RF-Non- 
RF Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Facilitate grade-level 
meetings 


22.2% 


3.72% 


0.000 


28.4% 


4.28% 


0.000 


6.3% 


4.57% 


0.192 


Coach staff on a range of 
topics 


7.2 


2.74 


0.009 


23.0 


3.58 


0.000 


15.8 


3.94 


0.000 


Provide direct reading 
instruction to students 


-21.6 


3.82 


0.000 


-20.6 


4.28 


0.000 


1.1 


4.69 


0.822 


Organize professional 
development for K-3 
teachers 


21.1 


3.35 


0.000 


25.1 


4.04 


0.000 


4.0 


4.32 


0.356 


Provide 

training/professional 
development in reading 


6.9 


2.44 


0.005 


20.7 


3.67 


0.000 


13.8 


3.83 


0.000 



materials, strategies, and 
assessments 

Source: Reading Coach Survey (2005); Question B4; Reading Coach Survey (2007); Question B4. 
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Exhibit B.6.3 

Percentage of Reading Coaches’ Rating Various Administrative and Schooi Support Activities as 
Centrai to Their Work in RF and Non-RF Titie i Schoois, 2004-05 and 2006-07 Schooi Years 

2004-05 2006-07 2006-07 - 2004-05 

Difference 





RF-Title 1 


Standard 




RF-Title 1 


Standard 




in 


Standard 




Activity 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Administer/coordinate 
reading assessments 


21.0% 


3.33% 


0.000 


17.1% 


3.80% 


0.000 


-3.9% 


4.82% 


0.422 


Compile reading 
assessment data 


20.7 


3.44 


0.000 


18.9 


3.69 


0.000 


-1.8 


4.21 


0.668 


Facilitate or coordinate 
family literacy activities 


1.7 


3.69 


0.645 


-1.5 


4.31 


0.767 


-3.2 


4.99 


0.525 


Order/manage reading 
instruction materials 


6.2 


3.43 


0.069 


13.8 


4.17 


0.001 


7.5 


4.60 


0.102 


Provide sub time for 
teachers to observe 
other more experienced 
teachers 


9.4 


3.53 


0.008 


21.2 


3.62 


0.000 


11.8 


4.46 


0.008 


Participate in school 
leadership team 
meetings 


2.3 


3.32 


0.451 


10.8 


3.84 


0.005 


8.6 


4.57 


0.062 


Participate in 
professional 
development provided 


7.0 


2.14 


0.001 


8.1 


3.02 


0.007 


1.1 


3.61 


0.751 



by the district, state or 
other consultants 

Source: Reading Coach Survey (2005); Question B4; Reading Coach Survey (2007); Question B4. 
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Exhibit B.6.4 



Percentage of Reading Coaches Rating Various Activities Supporting Teachers’ Instruction as 
Central to Their Work in RF and Non-RF Title I Schools, 2004-05 and 2006-07 School Years 







2004-05 






2006-07 




2006 


-07 - 2004 


-05 
















Difference 








RF-Title 1 


Standard 




RF-Title 1 


Standard 




in 


Standard 




Activity 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Give demonstration lessons 
with core/supplemental 
materials 


2.2% 


3.49% 


0.527 


19.3% 


4.22% 


0.000 


17.1% 


4.76% 


0.000 


Assist teachers in using the 
core reading program 


11.6 


3.05 


0.000 


29.2 


3.93 


0.000 


17.6 


4.48 


0.000 


Observe and provide 
feedback to teachers 


18.4 


3.37 


0.000 


27.9 


3.96 


0.000 


9.6 


4.31 


0.026 


Assist teachers in forming 
instructional groups 


15.4 


3.42 


0.000 


20.9 


4.01 


0.000 


5.5 


4.80 


0.249 


Help teachers design 
strategies for struggling 
readers 


4.6 


2.59 


0.077 


12.1 


3.13 


0.000 


7.5 


3.72 


0.045 


Help teachers monitor the 
effectiveness of strategies 
for struggling readers 


4.2 


2.84 


0.136 


15.9 


3.54 


0.000 


11.7 


4.35 


0.007 


Give demonstrations on 

assessment 

administration/scoring 


23.4 


3.62 


0.000 


10.1 


4.21 


0.019 


-13.3 


5.10 


0.009 


Plan reading instruction 
with teachers 


8.4 


3.71 


0.024 


20.1 


4.30 


0.000 


11.7 


4.76 


0.014 


Review teachers’ lesson 
plans & provide feedback 


9.0 


3.33 


0.007 


10.7 


3.71 


0.004 


1.6 


4.50 


0.714 


Help teachers in 
interpreting assessment 


19.8 


3.13 


0.000 


22.8 


3.51 


0.000 


2.9 


4.24 


0.487 



results 

Source: Reading Coach Survey (2005); Question B5; Reading Coach Survey (2007); Question B5. 
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Exhibit B.6.5 



Direct Support for Reading Instruction Received by Teachers in RF and Non-RF Title I Schools, 
2004-05 and 2006-07 School Years 







2004-05 






2006-07 




2006-07 


- 2004-05 


Topics 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference in 
Difference 


Standard 

Error 


P- 

value 


Coaching by reading coach in 
programs, materials or 
strategies 


36.9% 


1.76% 


0.000 


40.1% 


1.94% 


0.000 


3.2% 


2.16% 


0.141 


Coaching from fellow teacher 


10.7 


1.70 


0.000 


10.2 


1.90 


0.000 


-0.5 


2.24 


0.838 


Peer study group for group 
study 


20.8 


1.86 


0.000 


18.0 


2.06 


0.000 


-2.9 


2.28 


0.208 


Demonstrations in my 
classroom 


17.2 


1.97 


0.000 


17.2 


2.12 


0.000 


0.0 


2.41 


0.990 


Observations of other teachers 


6.1 


1.81 


0.001 


8.7 


1.93 


0.000 


2.6 


2.26 


0.251 


Diagnostic testing help from a 
reading coach or specialist for 
individual students 


27.5 


1.73 


0.000 


24.7 


2.02 


0.000 


-2.8 


2.28 


0.225 


Intervention service help from a 
reading coach or specialist for 
individual students 


17.8 


1.74 


0.000 


21.0 


1.98 


0.000 


3.2 


2.30 


0.169 


Interpretation of assessment 
data 


22.4 


1.46 


0.000 


21.0 


1.51 


0.000 


-1.4 


1.88 


0.471 


Grade level meetings devoted 
to reading 


15.4 


1.58 


0.000 


17.0 


1.61 


0.000 


1.6 


1.93 


0.402 


Using assessment data to 
determine topics that require 


14.7 


1.35 


0.000 


15.7 


1.41 


0.000 


1.0 


1.76 


0.584 



additional instruction 

Source: Teacher Survey (Question D2 in 2004-05, Question C2 in 2006-07). 
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Exhibit B.6.6 



Direct Support for Reading Instruction Not Available to Teachers in RF and Non-RF Title I Schools, 
2004-05 and 2006-07 School Years 







2004-05 






2006-07 




2006- 


07 - 2004- 


05 


Topics 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Coaching by reading coach in 
programs, materials or 
strategies 


-32.8% 


1.62% 


0.000 


-30.1% 


1.89% 


0.000 


2.7% 


2.08% 


0.202 


Coaching from fellow teacher 


-8.8 


1.53 


0.000 


-13.7 


1.74 


0.000 


-4.8 


1.98 


0.014 


Peer study group for group 
study 


-19.7 


1.96 


0.000 


-17.1 


2.09 


0.000 


2.5 


2.36 


0.280 


Demonstrations in my 
classroom 


-24.7 


1.94 


0.000 


-29.2 


2.01 


0.000 


-4.5 


2.15 


0.037 


Observations of other teachers 


-10.0 


1.77 


0.000 


-18.8 


1.93 


0.000 


-8.8 


2.19 


0.000 


Diagnostic testing help from a 
reading coach or specialist for 
individual students 


-24.0 


1.49 


0.000 


-22.1 


1.66 


0.000 


1.9 


1.86 


0.307 


Intervention service help from a 
reading coach or specialist for 
individual students 


-19.4 


1.47 


0.000 


-20.9 


1.59 


0.000 


-1.4 


1.85 


0.437 


Interpretation of assessment 
data 


-14.0 


1.08 


0.000 


-14.7 


1.19 


0.000 


-0.7 


1.38 


0.636 


Grade level meetings devoted 
to reading 


-12.1 


1.40 


0.000 


-14.8 


1.45 


0.000 


-2.7 


1.71 


0.109 


Using assessment data to 
determine topics that require 


-11.9 


1.14 


0.000 


-12.7 


1.14 


0.000 


-0.8 


1.46 


0.583 



additional instruction 

Source: Teacher Survey (Question D2 in 2004-05, Question C2 in 2006-07). 
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Responsibiiity for Oversight of Reading Activities in RF and Non-RF Titie i Schoois, 2004-05 and 


2006-07 Schooi Years 




















Reading Oversight 


2004-05 






2006-07 




2006-07 - 2004- 


-05 














Difference 






Activity/ 


RF-Title 1 


Standard 




RF-Title 1 


Standard 




in 


Standard 




Responsibie Party 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Monitoring impiementation of reading program 


...School alone is 
responsible 


-16.3% 


2.99% 


0.000 


-21.1% 


3.27% 


0.000 


-4.7% 


3.97% 


0.234 


...School and district are 
jointly responsible 


-16.2 


2.87 


0.000 


-14.5 


3.20 


0.000 


1.7 


3.89 


0.661 


...School, district and state 
are jointly responsible 


34.5 


2.32 


0.000 


36.3 


2.55 


0.000 


1.8 


3.17 


0.575 


. . .Other combinations 


-2.0 


0.90 


0.025 


-0.8 


0.58 


0.178 


1.2 


1.08 


0.248 


Seiection of reading professionai 
deveiopment topics and opportunities 


...School alone is 
responsible 


0.0 


2.94 


0.988 


-2.5 


3.29 


0.442 


-2.5 


4.03 


0.534 


...School and district are 
jointly responsible 


-21.5 


3.04 


0.000 


-9.6 


2.29 


0.006 


11.9 


4.12 


0.004 


...School, district and state 
are jointly responsible 


25.5 


2.07 


0.000 


14.7 


1.98 


0.000 


-10.8 


2.61 


0.000 


. . .Other combinations 


-4.0 


1.55 


0.011 


-2.6 


1.71 


0.087 


1.3 


2.13 


0.532 


Review of Teachers’ reading iesson pians 


...School alone is 
responsible 


-1.2 


1.48 


0.418 


-2.4 


1.87 


0.197 


-1.2 


2.01 


0.563 


...School and district are 
jointly responsible 


-1.2 


1.35 


0.362 


-0.2 


1.76 


0.942 


1.1 


1.78 


0.569 


...School, district and state 
are jointly responsible 


2.5 


0.62 


0.000 


2.5 


0.70 


0.000 


0.1 


0.86 


0.953 


. . .Other combinations 


0.0 


0.23 


0.842 


0.0 


0.00 


0.000 


0.5 


0.23 


0.841 


Feedback to teachers about reading instruction 


...School alone is 
responsible 


-2.6 


2.80 


0.345 


1.3 


3.06 


0.703 


3.9 


3.64 


0.279 


...School and district are 
jointly responsible 


-5.1 


2.50 


0.041 


-5.6 


2.78 


0.048 


-0.5 


3.37 


0.886 


...School, district and state 
are jointly responsible 


9.1 


1.59 


0.000 


4.9 


1.74 


0.004 


-4.2 


2.17 


0.054 


. . .Other combinations 


-1.4 


0.51 


0.005 


-0.6 


0.37 


0.082 


0.7 


0.58 


0.213 


interpretation of assessment resuits 


...School alone is 
responsible 


5.1 


3.04 


0.082 


-1.2 


3.33 


0.709 


-6.3 


4.03 


0.116 


...School and district are 
jointly responsible 


-11.3 


2.98 


0.000 


-11.5 


3.34 


0.001 


-0.2 


4.23 


0.964 


...School, district and state 
are jointly responsible 


9.8 


2.31 


0.000 


14.1 


2.59 


0.000 


4.2 


3.17 


0.184 


. . .Other combinations 


-3.6 


0.85 


0.000 


-1.3 


0.70 


0.060 


2.3 


1.04 


0.025 


Review individuai students’ progress in 


















reading 




















...School alone is 
responsible 


-3.2 


2.90 


0.277 


-8.2 


3.26 


0.014 


-4.9 


4.08 


0.226 


...School and district are 
jointly responsible 


-3.4 


2.54 


0.201 


-2.0 


3.03 


0.468 


1.4 


3.80 


0.717 


...School, district and state 
are jointly responsible 


8.8 


1.85 


0.000 


11.1 


1.92 


0.000 


1.8 


2.56 


0.475 


. . .Other combinations 


-2.2 


0.69 


0.001 


0.5 


0.36 


0.159 


1.7 


0.73 


0.017 


Source: Principal Survey (2005); Question D5; Principal Survey (2007); Question D3. 
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Exhibit B.6.8 



Percentage of RF and Non-RF Title I Schools in Which K-3 Reading Instruction Was Evaluated by 
the Principal, by Mode and Frequency of Evaluation, 2004-05 and 2006-07 School Years 







2004-05 






2006-07 




2006-07 - 2004 


-05 


Mode of Evaluation / 
Frequency of Evaluation 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Observed classroom reading 
instruction informally 


...Not at all 


-0.1% 


0.00% 


0.003 


0.2% 


0.00% 


0.318 


0.4% 


0.00% 


0.281 


...1-4 times this year 


-6.4 


-0.06 


0.020 


-6.7 


0.02 


0.001 


-0.2 


0.03 


0.950 


...5-8 times this year 


-3.5 


-0.04 


0.021 


-7.4 


0.03 


0.005 


-4.0 


0.03 


0.213 


...Once a month 


-7.9 


-0.08 


0.024 


1.2 


0.02 


0.621 


9.0 


0.03 


0.005 


. . .Once a week or more 


18.0 


0.18 


0.030 


12.7 


0.03 


0.000 


-5.3 


0.04 


0.181 


Observed classroom reading 
instruction using an evaluation form 


...Not at all 


-1.6 


-0.02 


0.014 


-3.8 


0.01 


0.003 


-2.0 


0.02 


0.232 


...1-4 times this year 


-8.6 


-0.09 


0.030 


-7.6 


0.03 


0.020 


1.2 


0.04 


0.752 


...5-8 times this year 


-1.3 


-0.01 


0.022 


3.8 


0.03 


0.178 


4.8 


0.03 


0.158 


...Once a month 


7.4 


0.07 


0.025 


3.5 


0.03 


0.222 


-4.0 


0.03 


0.254 


. . .Once a week or more 


4.1 


0.04 


0.023 


4.1 


0.03 


0.121 


-0.1 


0.03 


0.976 



Met with teachers individually to 
discuss strategies for improving 
reading instruction 



...Not at all 


-1.7 


-0.02 


0.013 


-0.9 


0.01 


0.450 


0.9 


0.01 


0.529 


...1-4 times this year 


-7.4 


-0.07 


0.029 


-7.2 


0.03 


0.024 


0.3 


0.04 


0.929 


...5-8 times this year 


1.0 


0.01 


0.026 


-1.5 


0.03 


0.612 


-2.6 


0.04 


0.501 


...Once a month 


4.3 


0.04 


0.027 


4.8 


0.03 


0.109 


0.5 


0.04 


0.892 


. . .Once a week or more 


3.8 


0.04 


0.023 


4.7 


0.03 


0.082 


0.8 


0.03 


0.809 



Met with groups of teachers to discuss 
strategies for improving reading 
instruction 



...Not at all 


0.3 


0.00 


0.010 


0.2 


0.01 


0.805 


0.0 


0.01 


0.985 


...1-4 times this year 


-9.6 


-0.10 


0.025 


-11.2 


0.03 


0.000 


-1.4 


0.03 


0.696 


...5-8 times this year 


-2.4 


-0.02 


0.026 


0.4 


0.03 


0.891 


2.6 


0.04 


0.497 


...Once a month 


6.7 


0.07 


0.029 


6.7 


0.03 


0.037 


0.0 


0.04 


0.997 


. . .Once a week or more 


5.1 


0.05 


0.023 


3.9 


0.03 


0.167 


-1.3 


0.03 


0.702 



Source: Principal Survey (2005); Question D6; Principal Survey (2007); Question D4. 
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Exhibit B.7.1 

Types of Reading-Reiated Professionai Deveiopment Activities Attended by Teachers in RF and 



Non-RF Titie i Schoois, 


2004-05 and 2006- 


■07 Schooi Years 
















2004-05 






2006-07 




2006 


-07 - 2004-05 
















Difference 






Type of Professional 


RF-Title I 


Standard 




RF-Title 1 


Standard 




in 


Standard 




Development Activity 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Difference 


Error 


p-value 


Half-day workshops 


13.6% 


1.78% 


0.000 


15.0% 


2.36% 


0.000 


1.3% 


2.74% 


0.625 


Full-day workshops 


29.7 


1.71 


0.000 


20.9 


2.36 


0.000 


-8.8 


2.74 


0.001 


Any workshop 


12.3 


1.05 


0.000 


17.1 


1.83 


0.000 


4.8 


1.99 


0.016 


College courses 


4.5 


1.37 


0.001 


4.1 


1.60 


0.009 


-0.3 


1.98 


0.862 


Conferences 


9.8 


1.68 


0.000 


12.2 


2.00 


0.000 


2.4 


2.31 


0.302 






Mean 






Mean 






Mean 






3.7 


0.22 


0.000 


2.6 


0.21 


0.000 


-1.1 


0.26 


0.000 


Number of total hours in 




















attendance across all 




















activities 


24.0 


1.37 


0.000 


15.6 


1.50 


0.000 


-8.4 


1.86 


0.000 


Source: Teacher Survey (Question D1 in 2004-05, Question Cl in 2006-07). 



Exhibit B.7.2 



Design Features of the Professionai Deveiopment Activities Attended by Teachers in RF and in Non- 
RF Titie i Schoois, 2004-05 and 2006-07 Schooi Years 







2004-05 






2006-07 




2006- 


-07 - 2004- 


-05 


Professional development 
activities that: 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Are given by trainers or 
facilitators who have a well- 
established reputation 


14.6% 


1.31% 


0.000 


14.5% 


1.51% 


0.000 


-0.1% 


1.66% 


0.959 


Are held in a convenient 
location (e.g., activities held 
at school) 


12.9 


1.40 


0.000 


10.7 


1.61 


0.000 


-2.2 


1.83 


0.227 


Use a team-based approach 
(joint training of people who 
work together) 


20.1 


1.37 


0.000 


16.4 


1.52 


0.000 


-3.7 


1.78 


0.038 


Were also attended by the 
principal 


13.9 


1.58 


0.000 


8.2 


1.78 


0.000 


-5.7 


1.97 


0.004 


Provide follow-up activities 


16.1 


1.44 


0.000 


11.5 


1.55 


0.000 


-4.6 


1.80 


0.011 


Include release time for 
participating teachers 


13.5 


1.54 


0.000 


11.3 


1.67 


0.000 


-2.2 


1.97 


0.263 


Provide a stipend? 


25.9 


1.47 


0.000 


17.4 


1.69 


0.000 


-8.6 


1.77 


0.000 


Provide teachers options 
among which to choose 


-1.8 


1.29 


0.172 


0.2 


1.62 


0.948 


1.9 


1.84 


0.290 


Offer graduate college credits 


6.3 


1.23 


0.000 


3.6 


1.35 


0.009 


-2.6 


1.56 


0.093 



Source: Teacher Survey (Question D3 in 2004-05, Question C3 in 2006-07). 
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Exhibit B.7.3 

Teacher Participation in Professionai Deveiopment Activities Reiated to the Five Dimensions of 
Reading: RF and Non-RF Titie I Schoois, 2004-05 and 2006-07 Schooi Years 

2004-05 2006-07 2006-07 - 2004-05 

Difference 



Dimension 


RF-Titie i 
Difference 


Standard 

Error 


p-vaiue 


RF-Titie i 
Difference 


Standard 

Error 


p-vaiue 


in 

Difference 


Standard 

Error 


p-vaiue 


Phonemic Awareness 


28.7% 


1.48% 


0.000 


24.2% 


1.63% 


0.000 


-4.5% 


2.02% 


0.027 


Phonics 


27.8 


1.51 


0.000 


23.9 


1.61 


0.000 


-3.9 


2.01 


0.054 


Vocabulary 


26.3 


1.66 


0.000 


22.3 


1.75 


0.000 


-4.1 


2.20 


0.064 


Fluency 


20.4 


1.60 


0.000 


16.9 


1.76 


0.000 


-3.5 


2.10 


0.238 


Comprehension 


13.8 


1.46 


0.000 


14.5 


1.59 


0.000 


-0.7 


1.94 


0.727 



Source: Teacher Survey (Question D4 in 2004-05, Question C4 in 2006-07). 



Exhibit B.7.4 

Teachers’ Perceived Professional Development Needs Related to the Five Dimensions of Reading: 
RF and Non-RF Title I Schools, 2004-05 and 2006-07 School Years 

2004-05 2006-07 2006-07 - 2004-05 

Difference 



Dimension 


RF-Titie i 
Difference 


Standard 

Error 


p-vaiue 


RF-Titie i 
Difference 


Standard 

Error 


p-vaiue 


in 

Difference 


Standard 

Error 


p-vaiue 


Phonemic Awareness 


-9.9% 


1.82% 


0.000 


-19.1% 


2.35% 


0.000 


-9.2% 


2.86% 


0.001 


Phonics 


-3.5 


1.79 


0.052 


-12.9 


2.54 


0.000 


-9.4 


3.04 


0.002 


Vocabulary 


7.4 


1.79 


0.000 


4.2 


2.46 


0.094 


-3.2 


2.98 


0.286 


Fluency 


0.6 


2.19 


0.793 


-10.5 


2.71 


0.000 


-11.0 


3.38 


0.001 


Comprehension 


1.2 


1.58 


0.434 


-6.4 


2.25 


0.005 


-7.6 


2.66 


0.004 



Source: Teacher Survey (Question D4 in 2004-05, Question C4 in 2006-07). 



Exhibit B.7.5 

Preparedness to Teach Five Dimensions of Reading: Teachers in RF and Non-RF Title I Schools, 
2004-05 and 2006-07 School Years 

2004-05 2006-07 2006-07 - 2004-05 

Difference 



Dimension 


RF-Titie i 
Difference 


Standard 

Error 


p-vaiue 


RF-Titie i 
Difference 


Standard 

Error 


p-vaiue 


in 

Difference 


standard 

Error 


p-vaiue 


Phonemic Awareness 


0.5 


0.05 


0.000 


0.5 


0.05 


0.000 


0.0 


0.06 


0.848 


Phonics 


0.5 


0.04 


0.000 


0.6 


0.05 


0.000 


0.0 


0.06 


0.463 


Vocabulary 


0.4 


0.04 


0.000 


0.5 


0.05 


0.000 


0.1 


0.06 


0.126 


Fluency 


0.3 


0.04 


0.000 


0.4 


0.05 


0.000 


0.1 


0.06 


0.082 


Comprehension 


0.4 


0.04 


0.000 


0.5 


0.05 


0.000 


0.1 


0.05 


0.022 



Source: Teacher Survey (Question D6 in 2004-05, Question C6 in 2006-07). 



B-40 



Appendix B 









Exhibit B.7.6 

Professional Development Related to the Five Dimensions of Reading Attended by Principals and 
Reading Coaches in RF and non-RF Title I Schools, 2004-05 and 2006-07 School Years 





2004-05 




2006-07 


2006-07 - 2004-05 


RF-Title I 


Standard 


RF-Title I 


Standard 


Difference 

in Standard 


Difference 


Error 


p-value Difference 


Error 


p-value Difference Error p-value 



Principal Participation 


Phonemic Awareness 


28.8% 


2.56% 


0.000 


18.1% 


3.06% 


0.000 


-1 1 .0% 


3.61% 


0.003 


Phonics 


30.5 


2.68 


0.000 


20.9 


3.11 


0.000 


-9.6 


3.80 


0.012 


Vocabulary 


25.4 


2.55 


0.000 


14.9 


2.80 


0.000 


-10.5 


3.55 


0.003 


Comprehension 


17.3 


2.43 


0.000 


13.8 


2.80 


0.000 


-3.4 


3.59 


0.340 


Fluency 


26.7 


2.47 


0.000 


19.3 


2.91 


0.000 


-7.4 


3.57 


0.038 


Reading Coach Participation 


Essential Components of 
Reading 


33.7 


3.08 


0.000 


14.9 


3.46 


0.000 


-18.7 


4.53 


0.000 



Source: Principal survey (Question F2 a-e in 2004-05, Question FI a-e in 2006-07), Reading Coach Survey (Question Gif in 2004- 
05,Flfin 2006-07). 



Exhibit B.7.7 



Professional Development Related to Other Features of Reading Instruction for Teachers in RF and 
Non-RF Title I Schools, 2004-05 and 2006-07 School Years 



Area of Professional 
Development 




2004-05 






2006-07 




2006 


-07 - 2004- 


-05 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Teaching strategies 


8.2% 


0.89% 


0.000 


6.3% 


1.15% 


0.000 


-1.9% 


1.39% 


0.167 


Grouping 


13.4 


1.67 


0.000 


10.0 


1.72 


0.000 


-3.4 


2.20 


0.123 


Assessment 


18.1 


1.49 


0.000 


10.3 


1.50 


0.000 


-7.7 


2.03 


0.000 


Struggling readers 


12.1 


1.60 


0.000 


13.8 


1.66 


0.000 


1.7 


2.14 


0.438 


Organization/planning 


2.1 


1.40 


0.127 


3.5 


1.48 


0.019 


1.4 


1.87 


0.452 


Number of Professional Development Activities 


Teaching strategies 


1.1 


0.08 


0.000 


0.8 


0.11 


0.000 


-0.3 


0.12 


0.010 


Grouping 


0.2 


0.03 


0.000 


0.2 


0.04 


0.000 


0.0 


0.04 


0.365 


Assessment 


0.6 


0.04 


0.000 


0.4 


0.05 


0.000 


-0.2 


0.06 


0.000 


Struggling readers 


0.8 


0.07 


0.000 


0.8 


0.09 


0.000 


0.0 


0.11 


0.914 


Organization/planning 


0.2 


0.05 


0.000 


0.2 


0.07 


0.001 


0.0 


0.08 


0.975 



Source: Teacher Survey (Question D5 in 2004-05, Question C5 in 2006-07). 
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Exhibit B.7.8 



Teachers’ Perceived Need for Additional Professional Development Related to Other Features of 
Reading Instruction in RF and Non-RF Title I Schools, 2004-05 and 2006-07 School Years 



Area of Professional 
Development 




2004-05 






2006-07 




2006 


-07 - 2004- 


-05 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Teaching strategies 


3.4% 


1.45% 


0.020 


-0.2 


1.63% 


0.885 


-3.5% 


2.10% 


0.092 


Grouping 


1.7 


1.70 


0.319 


-2.3 


2.13 


0.266 


-4.0 


2.60 


0.122 


Assessment 


-5.1 


1.69 


0.002 


-8.9 


2.12 


0.000 


-3.8 


2.56 


0.139 


Struggling readers 


-3.6 


1.25 


0.004 


-2.1 


1.79 


0.232 


1.5 


2.14 


0.484 


Organization/planning 


6.3 


1.74 


0.000 


2.6 


2.15 


0.236 


-3.7 


2.62 


0.159 



Source: Teacher Survey (Question D5 in 2004-05, Question C5 in 2006-07). 



Exhibit B.7.9 




















Professionai Deveiopment Reiated to Other Features of Reading instruction Attended by Principais 
in RF and Non-RF Titie i Schoois, 2004-05 and 2006-07 Schooi Years 






2004-05 






2006-07 




2006 


-07 - 2004-05 


Area of Professional 
Development 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


RF-Title 1 
Difference 


Standard 

Error 


p-value 


Difference 

in 

Difference 


Standard 

Error 


p-value 


Teaching strategies 


1 1 .4% 


2.08% 


0.000 


6.9% 


2.25% 


0.002 


-4.5% 


2.89% 


0.121 


Grouping 


22.0 


2.78 


0.000 


18.9 


3.03 


0.000 


-3.2 


3.90 


0.415 


Assessment 


13.6 


2.42 


0.000 


6.1 


2.48 


0.015 


-7.5 


3.21 


0.019 


Struggling readers 


12.4 


2.86 


0.000 


7.3 


3.07 


0.016 


-5.1 


4.01 


0.206 


Organization/planning 


2.9 


2.44 


0.239 


-0.3 


2.67 


0.916 


-3.2 


3.43 


0.349 


Number of Professional 
Development Activities 


Teaching strategies 


1.2 


0.14 


0.000 


0.6 


0.15 


0.000 


-0.6 


0.18 


0.002 


Grouping 


0.8 


0.09 


0.000 


0.6 


0.10 


0.000 


-0.2 


0.12 


0.053 


Assessment 


0.5 


0.07 


0.000 


0.2 


0.08 


0.018 


-0.3 


0.10 


0.000 


Struggling readers 


0.8 


0.12 


0.000 


0.3 


0.14 


0.038 


-0.5 


0.16 


0.002 


Organization/planning 


0.4 


0.12 


0.006 


0.2 


0.15 


0.101 


-0.1 


0.17 


0.545 


Source: Principal Survey (Question F2 in 2004-05, Question FI in 2006-07). 
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Exhibit B.7.10 



Professional Development in Teaching Strategies Attended by Reading Coaches in RF and Non-RF 
Title I Schools, 2004-05 and 2006-07 School Years 




Appendix B 



B-43 












Exhibit B.8.1 



Years of State Test Score Data Included in Achievement Analyses, by State 







Third Grade 






Fourth Grade 






Number of 


First year 


Last year 


Number of 


First 


Last year 




years of 


of test 


of test 


years of 


year of 


of test 


State 


test data 


data 


data 


test data 


test data 


data 


Arkansas 








3 


2004 


2006 


Arizona 


7 


2000 


2006 








California 


4 


2003 


2006 








Colorado 


9 


1998 


2006 


3 


2004 


2006 


Connecticut 








6 


2001 


2006 


District of Columbia 


3 


2004 


2006 








Delaware 


9 


1998 


2006 








Florida 


7 


2000 


2006 


3 


2004 


2006 


Georgia 


4 


2002 


2005 








Hawaii 


10 


1997 


2006 








Iowa 








4 


2003 


2006 


Idaho 








4 


2003 


2006 


Illinois 


8 


1999 


2006 








Indiana 


8 


1999 


2006 








Kentucky 


8 


1999 


2006 


8 


1999 


2006 


Louisiana 








7 


2000 


2006 


Massachusetts 


6 


2001 


2006 








Maryland 


4 


2003 


2006 








Michigan 








3 


2003 


2005 


Minnesota 


8 


1998 


2005 








Mississippi 


6 


2001 


2006 


3 


2004 


2006 


Montana 








6 


2001 


2006 


North Carolina 


9 


1998 


2006 








North Dakota 








4 


2003 


2006 


New Jersey 








4 


2003 


2006 


Nevada 


5 


2002 


2006 








Ohio 








3 


2004 


2006 


Oregon 


9 


1998 


2006 








South Carolina 


8 


1999 


2006 








South Dakota 


3 


2004 


2006 








Utah 








2 


2004 


2005 


Virginia 


9 


1998 


2006 








Washington 


6 


2000 


2005 


9 


1998 


2006 


Wisconsin 


2 


2004 


2005 


3 


2004 


2006 


West Virginia 


3 


2004 


2006 









Source: State Databases. 
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Exhibit B.8.2 



Estimated Pre- and Post-RF Third-Grade Reading Scores and Model Statistics Using Four 
Methods to Assign Award Dates 





RF Mean Scores 


Non-RF Title I Mean 
Scores 






Award 

Date 

State Method 


Post- 

Pre-RF RF 


Pre-RF Post-RF 


Difference in 
Difference 


Standard 
Error p-value 


Panel 1: States Reporting Scores as the Percent of Students Meeting a Proficiency Standard 



Arizona 


state 


80.1% 


87.6% 


83.9% 


86.2% 


5.05% 


1.01% 


0.000 


Arizona 


district 


81.7 


88.5 


84.4 


84.9 


6.30 


1.12 


0.000 


Arizona 


adj. district 


82.0 


87.6 


84.4 


85.0 


5.01 


1.06 


0.000 


Arizona 


school 


82.0 


87.2 


84.5 


85.0 


4.61 


1.05 


0.000 


California 


district 


16.3 


20.8 


25.7 


26.9 


3.25 


0.41 


0.000 


California 


adj. district 


16.5 


20.4 


25.6 


27.0 


2.60 


0.38 


0.000 


California 


school 


16.5 


19.5 


25.7 


27.0 


1.60 


0.34 


0.000 


Colorado 


state 


87.7 


83.8 


89.0 


84.8 


0.28 


2.67 


0.918 


Colorado 


district 


88.0 


82.6 


89.8 


86.3 


-1.82 


2.20 


0.409 


Colorado 


adj. district 


88.0 


82.6 


89.8 


86.3 


-1.83 


2.20 


0.405 


Colorado 


school 


88.0 


82.6 


89.8 


86.3 


-1.83 


2.20 


0.406 


D.C. 


district 


37.9 


31.0 


51.5 


46.2 


-1.46 


5.74 


0.799 


D.C. 


adj. district 


37.9 


31.0 


51.5 


46.2 


-1.46 


5.74 


0.799 


D.C. 


school 


37.9 


31.0 


51.5 


46.2 


-1.46 


5.74 


0.799 


Georgia 


state 


74.8 


85.7 


79.1 


88.7 


1.23 


1.04 


0.239 


Georgia 


district 


79.5 


80.4 


83.2 


88.7 


-4.68 


9.51 


0.623 


Georgia 


adj. district 


79.5 


90.1 


83.2 


88.7 


5.04 


7.30 


0.491 


Georgia 


school 


79.4 


91.0 


83.2 


88.7 


6.00 


4.22 


0.115 


Hawaii 


state 


97.4 


82.2 


94.6 


79.0 


0.36 


3.27 


0.913 


Hawaii 


district 


90.6 


78.9 


87.3 


79.3 


-3.70 


2.20 


0.094 


Hawaii 


adj. district 


90.6 


78.9 


87.3 


79.3 


-3.70 


2.20 


0.094 


Hawaii 


school 


89.2 


82.1 


87.4 


79.0 


1.27 


1.99 


0.522 


Illinois 


state 


34.3 


46.0 


43.5 


52.0 


3.18 


0.90 


0.000 


Illinois 


district 


38.0 


47.7 


48.9 


51.2 


7.35 


1.46 


0.000 


Illinois 


adj. district 


39.9 


46.9 


49.1 


51.2 


4.86 


1.30 


0.000 


Illinois 


school 


40.0 


46.9 


49.2 


51.2 


4.83 


1.31 


0.000 


Indiana 


state 


58.2 


65.0 


54.6 


61.6 


-0.11 


2.81 


0.970 


Indiana 


district 


57.0 


65.0 


54.7 


61.3 


1.42 


2.89 


0.623 


Indiana 


adj. district 


57.1 


65.0 


54.7 


61.3 


1.29 


2.89 


0.654 


Indiana 


school 


57.1 


65.0 


54.7 


61.3 


1.29 


2.89 


0.654 


Maryland 


state 


34.7 


62.7 


42.0 


65.8 


4.24 


2.29 


0.064 


Maryland 


district 


44.0 


65.6 


49.6 


65.8 


5.33 


2.42 


0.028 


Maryland 


adj. district 


44.0 


65.1 


49.6 


65.8 


4.90 


2.42 


0.043 


Maryland 


school 


44.0 


65.2 


49.6 


65.8 


5.00 


2.42 


0.039 


Massachusetts 


state 


37.9 


37.0 


44.2 


40.7 


2.53 


1.57 


0.108 


Massachusetts 


district 


38.1 


35.1 


44.3 


41.0 


0.43 


1.54 


0.781 


Massachusetts 


adj. district 


38.1 


35.2 


44.3 


41.0 


0.53 


1.54 


0.730 


Massachusetts 


school 


37.9 


35.8 


44.2 


41.3 


0.80 


1.47 


0.587 


Mississippi 


state 


57.8 


81.9 


75.0 


83.3 


15.84 


1.74 


0.000 


Mississippi 


district 


61.3 


84.2 


76.7 


83.2 


16.43 


1.84 


0.000 


Mississippi 


adj. district 


61.3 


84.3 


76.7 


83.2 


16.60 


1.84 


0.000 


Mississippi 


school 


61.3 


84.4 


76.7 


83.2 


16.62 


1.84 


0.000 



(Continues) 
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Exhibit B.8.2 



Estimated Pre- and Post-RF Third-Grade Reading Scores and Model Statistics Using Four 
Methods to Assign Award Dates (Continued) 







RF Mean Scores 


Non-RF Title I Mean 
Scores 






State 


Award 

Date 

Method 


Post- 

Pre-RF RF 


Pre-RF Post-RF 


Difference in 
Difference 


Standard 
Error p-value 



Panel 1: States Reporting Scores as the Percent of Students Meeting a Proficiency Standard 



Nevada 


state 


87.1% 


86.1% 


80.3% 


81.5% 


-2.24% 


1.71% 


0.191 


Nevada 


district 


87.7 


86.5 


81.4 


81.9 


-1.82 


1.68 


0.279 


Nevada 


adj. district 


87.7 


86.3 


81.4 


81.9 


-1.97 


1.70 


0.248 


Nevada 


school 


87.8 


86.2 


81.3 


82.1 


-2.38 


1.58 


0.132 


North Carolina 


state 


69.2 


71.0 


82.9 


79.5 


5.14 


1.39 


0.000 


North Carolina 


district 


71.8 


71.7 


82.9 


79.5 


3.25 


1.32 


0.014 


North Carolina 


adj. district 


71.9 


71.4 


82.9 


79.5 


2.89 


1.31 


0.028 


North Carolina 


school 


71.9 


71.4 


82.9 


79.5 


2.89 


1.31 


0.028 


Oregon 


state 


81.8 


83.1 


92.1 


85.2 


8.18 


3.38 


0.016 


Oregon 


district 


78.7 


84.6 


86.7 


83.8 


8.77 


2.94 


0.003 


Oregon 


adj. district 


79.7 


83.8 


85.8 


83.8 


6.17 


2.48 


0.013 


Oregon 


school 


79.7 


83.8 


85.8 


83.8 


6.10 


2.47 


0.014 


South Carolina 


state 


79.1 


77.7 


93.5 


86.1 


5.95 


2.20 


0.007 


South Carolina 


district 


80.7 


75.9 


92.2 


85.9 


1.38 


1.94 


0.478 


South Carolina 


adj. district 


80.8 


75.7 


92.2 


85.9 


1.24 


1.93 


0.522 


South Carolina 


school 


80.8 


75.7 


92.2 


85.9 


1.24 


1.93 


0.520 


South Dakota 


district 


72.3 


87.6 


73.4 


79.1 


9.57 


4.66 


0.042 


South Dakota 


adj. district 


72.3 


87.7 


73.4 


79.1 


9.70 


4.64 


0.039 


South Dakota 


school 


72.3 


87.7 


73.4 


79.1 


9.68 


4.64 


0.039 


Virginia 


state 


58.3 


75.4 


76.2 


78.0 


15.22 


2.98 


0.000 


Virginia 


district 


65.1 


78.4 


78.1 


76.3 


15.15 


2.52 


0.000 


Virginia 


adj. district 


65.1 


78.4 


78.1 


76.3 


15.14 


2.52 


0.000 


Virginia 


school 


65.1 


78.4 


78.1 


76.3 


15.14 


2.52 


0.000 


Washington 


state 


58.3 


65.6 


75.2 


76.9 


5.60 


1.30 


0.000 


Washington 


district 


59.2 


64.6 


75.7 


76.8 


4.36 


1.13 


0.000 


Washington 


adj. district 


59.3 


64.1 


75.6 


76.8 


3.69 


1.11 


0.001 


Washington 


school 


59.1 


64.6 


75.6 


76.9 


4.21 


1.05 


0.000 


West Virginia 


district 


71.2 


77.7 


74.3 


76.1 


4.64 


2.35 


0.049 


West Virginia 


adj. district 


71.2 


77.8 


74.3 


76.1 


4.76 


2.34 


0.043 


West Virginia 


school 


71.2 


78.1 


74.5 


76.2 


5.17 


2.44 


0.035 


Wisconsin 


district 


91.0 


91.5 


90.6 


91.7 


-0.50 


1.13 


0.659 


Wisconsin 


adj. district 


91.0 


91.5 


90.6 


91.7 


-0.49 


1.12 


0.661 


Wisconsin 


school 


91.3 


90.4 


90.8 


91.9 


-1.93 


1.66 


0.246 



Panel 2: States Reporting Scores in Median Percentiles or Mean Scaled Scores 



Delaware 


state 


421.5 


432.7 


420.7 


439.7 


-7.75 


4.91 


0.118 


Delaware 


district 


421.5 


433.2 


420.7 


440.0 


-7.51 


5.39 


0.166 


Delaware 


adj. district 


421.5 


433.1 


420.7 


440.0 


-7.65 


5.40 


0.160 


Delaware 


school 


421.5 


433.1 


420.7 


440.0 


-7.65 


5.40 


0.160 


Florida 


state 


44.2 


45.4 


45.4 


47.0 


-0.49 


0.46 


0.295 


Florida 


district 


44.5 


46.9 


45.9 


48.6 


-0.29 


0.50 


0.559 


Florida 


adj. district 


44.6 


46.6 


45.9 


48.6 


-0.66 


0.48 


0.167 


(Continues) 
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Estimated Pre- and Post-RF Third-Grade Reading Scores and Model Statistics Using Four 
Methods to Assign Award Dates (Continued) 







RF Mean 
Scores 


Non-RF Title 1 Mean 
Scores 








State 


Award Date 
Method 


Pre- 

RF 


Post- 

RF 


Pre-RF 


Post-RF 


Difference in 
Difference 


Standard 

Error 


p-value 


Panel 2: States Reporting Scores in Median Percentiles or Mean Scaled Scores 


Florida 


school 


44.7 


46.5 


46.0 


48.6 


-0.81 


0.47 


0.116 


Kentucky 


state 


57.4 


54.3 


65.4 


60.6 


1.64 


1.71 


0.338 


Kentucky 


district 


59.3 


50.7 


65.3 


60.6 


-3.93 


1.59 


0.013 


Kentucky 


adj. district 


59.2 


51.0 


65.3 


60.6 


-3.51 


1.58 


0.027 


Kentucky 


school 


59.0 


51.7 


65.3 


60.6 


-2.64 


1.52 


0.082 


Minnesota 


state 


1352 


1449 


1326 


1427 


-4.98 


12.06 


0.680 


Minnesota 


district 


1371 


1449 


1345 


1426 


-1.89 


14.66 


0.898 


Minnesota 


adj. district 


1371 


1449 


1345 


1426 


-1.89 


14.66 


0.898 


Minnesota 


school 


1371 


1449 


1345 


1426 


-1.89 


14.66 


0.898 


Note: Shaded eells indieate that data were fit using a linear model, all other analyses used a baseline mean model. 

In Panel 2, the seores in Florida and Kentueky are reported as median pereentile ranks; in Delaware and Minnesota, the seores 
are mean sealed seores. 

Souree: State databases. 
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Exhibit B.8.3 



Estimated Pre- and Post-RF Fourth-Grade Reading Scores and Model Statistics Using Four 
Different Methods to Assign Award Dates 



RF Mean Scores 


Non-RF Title I 
Mean Scores 




Award Date 

state Method Pre-RF Post-RF 


Pre-RF Post-RF 


Difference 

in Standard p- 

Difference Error vaiue 


Panel 1: States Reporting Scores as the Percent of Students Meeting a Proficiency Standard 



Arkansas 


district 


81.0% 


79.0% 


86.4% 


84.7% 


-0.33% 


1.66% 


0.842 


Arkansas 


adj. district 


80.8 


79.2 


86.2 


84.8 


-0.27 


1.60 


0.865 


Arkansas 


school 


80.8 


79.2 


86.2 


84.8 


-0.28 


1.59 


0.863 


Colorado 


district 


76.8 


81.6 


81.0 


81.4 


4.32 


1.79 


0.016 


Colorado 


adj. district 


76.6 


81.7 


80.1 


82.4 


2.78 


1.58 


0.078 


Colorado 


school 


76.8 


80.2 


80.7 


81.7 


2.43 


1.71 


0.157 


Connecticut 


state 


57.5 


63.5 


54.4 


56.5 


3.97 


2.58 


0.124 


Connecticut 


district 


57.3 


63.5 


54.7 


56.5 


4.56 


2.47 


0.065 


Connecticut 


adj. district 


57.3 


63.5 


54.7 


56.5 


4.56 


2.47 


0.065 


Connecticut 


school 


57.6 


61.8 


54.7 


56.5 


2.48 


2.52 


0.325 


Idaho 


state 


92.1 


96.3 


92.2 


97.1 


-0.64 


1.31 


0.628 


Idaho 


district 


93.3 


96.6 


93.6 


96.2 


0.71 


0.96 


0.462 


Idaho 


adj. district 


93.3 


96.6 


93.6 


96.2 


0.72 


0.96 


0.454 


Idaho 


school 


93.3 


96.6 


93.6 


96.2 


0.69 


0.97 


0.473 


Iowa 


state 


61.6 


64.5 


63.3 


67.4 


-1.16 


2.16 


0.594 


Iowa 


district 


63.0 


65.6 


64.3 


68.6 


-1.75 


1.77 


0.324 


Iowa 


adj. district 


63.0 


65.5 


64.3 


68.6 


-1.76 


1.77 


0.321 


Iowa 


school 


63.0 


65.6 


64.3 


68.6 


-1.74 


1.77 


0.328 


Kentucky 


state 


78.0 


83.9 


84.0 


89.4 


0.57 


1.09 


0.602 


Kentucky 


district 


78.7 


84.0 


84.7 


89.4 


0.60 


1.09 


0.578 


Kentucky 


adj. district 


79.2 


83.0 


84.9 


89.5 


-0.80 


1.25 


0.523 


Kentucky 


school 


79.2 


83.9 


84.8 


89.5 


-0.05 


1.11 


0.967 


Louisiana 


state 


78.4 


79.9 


82.4 


81.6 


2.32 


1.31 


0.076 


Louisiana 


district 


79.2 


80.2 


82.8 


82.4 


1.37 


1.20 


0.257 


Louisiana 


adj. district 


79.2 


79.9 


82.9 


81.6 


2.07 


1.23 


0.091 


Louisiana 


school 


79.1 


80.1 


82.9 


81.6 


2.14 


1.21 


0.077 


Michigan 


district 


87.6 


92.1 


88.4 


92.0 


0.91 


0.85 


0.281 


Michigan 


adj. district 


88.0 


92.3 


88.8 


92.4 


0.75 


0.84 


0.374 


Michigan 


school 


87.9 


92.2 


88.8 


92.4 


0.77 


0.84 


0.357 


Mississippi 


district 


89.3 


92.4 


94.1 


93.8 


3.41 


1.23 


0.005 


Mississippi 


adj. district 


89.3 


92.5 


94.1 


93.8 


3.45 


1.23 


0.005 


Mississippi 


school 


89.3 


92.5 


94.1 


93.8 


3.49 


1.22 


0.005 


Montana 


state 


82.0 


85.4 


87.3 


87.0 


3.56 


2.90 


0.221 


Montana 


district 


82.2 


85.3 


87.1 


87.0 


3.08 


2.80 


0.273 


Montana 


adj. district 


82.2 


85.3 


87.1 


87.0 


3.08 


2.80 


0.273 


Montana 


school 


82.2 


85.3 


87.1 


87.0 


3.09 


2.80 


0.271 


New Jersey 


state 


53.5 


60.5 


57.1 


60.0 


4.05 


2.19 


0.065 


New Jersey 


district 


59.6 


60.8 


61.7 


60.3 


2.56 


1.70 


0.134 


New Jersey 


adj. district 


58.9 


60.1 


61.7 


60.1 


2.88 


1.79 


0.107 


New Jersey 


school 


58.2 


61.3 


61.6 


60.5 


4.21 


1.67 


0.012 



(Continues) 
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Exhibit B.8.3 



Estimated Pre- and Post-RF Fourth-Grade Reading Scores and Model Statistics Using Four 
Different Methods to Assign Award Dates (Continued) 







RF Mean Scores 


Non-RF Title I 
Mean Scores 








state 


Award Date 
Method 


Pre-RF Post-RF 


Pre-RF Post-RF 


Difference 

in 

Difference 


Standard 

Error 


P- 

vaiue 



Panel 1: States Reporting Scores as the Percent of Students Meeting a Proficiency Standard 



North Dakota 


state 


70.1% 


71.2% 


66.9% 


66.6% 


1.41% 


4.60% 


0.760 


North Dakota 


district 


71.2 


70.1 


64.2 


62.8 


0.30 


4.43 


0.946 


North Dakota 


adj. district 


71.2 


70.1 


64.2 


62.8 


0.29 


4.43 


0.948 


North Dakota 


school 


71.2 


69.9 


64.2 


62.8 


0.17 


4.42 


0.969 


Ohio 


district 


50.9 


52.5 


65.8 


69.7 


-2.24 


1.56 


0.153 


Ohio 


adj. district 


50.9 


52.5 


65.8 


69.7 


-2.22 


1.56 


0.156 


Ohio 


school 


51.0 


52.6 


65.8 


69.8 


-2.33 


1.55 


0.133 


Utah 


district 


69.5 


74.6 


78.3 


82.0 


1.32 


2.91 


0.653 


Utah 


adj. district 


69.5 


74.6 


78.3 


82.0 


1.32 


2.91 


0.653 


Utah 


school 


69.5 


74.6 


78.3 


82.0 


1.32 


2.91 


0.653 


Washington 


state 


83.0 


92.0 


88.8 


94.2 


3.66 


0.97 


0.000 


Washington 


district 


84.1 


91.3 


89.2 


93.0 


3.48 


0.86 


0.000 


Washington 


adj. district 


84.1 


90.5 


89.2 


93.0 


2.66 


0.83 


0.001 


Washington 


school 


84.1 


91.1 


89.3 


93.0 


3.29 


0.79 


0.000 


Wisconsin 


state 


86.0 


88.6 


90.0 


88.1 


4.51 


1.54 


0.004 


Wisconsin 


district 


86.6 


88.7 


89.2 


88.1 


3.17 


1.37 


0.021 


Wisconsin 


adj. district 


86.6 


88.8 


89.2 


88.1 


3.21 


1.36 


0.019 


Wisconsin 


school 


86.6 


88.6 


89.4 


88.0 


3.44 


1.29 


0.008 


Panel 2: States Reporting Scores in Median Percentiles or Mean Scaled Scores 


Florida 


district 


51.6 


59.6 


54.4 


56.8 


5.55 


0.67 


0.000 


Florida 


adj. district 


52.0 


58.7 


54.2 


57.2 


3.63 


0.64 


0.000 


Florida 


school 


52.1 


58.5 


54.1 


57.0 


3.56 


0.66 


0.000 


Note: In Panel 2, Florida’s seores are reported as median pereentile ranks. 










Souree: State databases. 
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Appendix C presents the statistical models that were used in the analyses that produced the 
findings presented in this report. First, we specify the statistical model used to analyze the 
survey data collected from principals, teachers and reading coaches in the 2004-05 and 2006-07 
school years. The model provides a set of estimates that allows us to a) compare the status of 
reading programs in RF and non-RF Title I schools, either in 2005 or in 2007, b) examine 
longitudinal change from 2005 to 2007 in RF and non-RF Title I schools, and c) compare the 
differential change between the two groups of schools. The results of these analyses are 
presented in Chapters 3 through 7. 

Second, we specify the statistical models used to conduct two analyses of student reading 
achievement. The first set of models are used to investigate trends in student reading 
achievement in RF and non-RF Title I schools using scores on third- and fourth-grade state 
reading assessments. The second model uses third-grade reading scores on 2005 state 
assessments (as used above); and selected responses from the spring 2005 teacher survey to 
investigate the relationship between student reading achievement and the implementation of RF 
activities. 

Analysis of 2004-05 and 2006-07 Survey Data 

The following model produces a series of parameter estimates that allow us to examine the 
status, the longitudinal change, and the differential change in reading programs in RF and non- 
RF Title I schools in 2005 and 2007. 

The model is specified as follows: 

Yu =Po+ PxRFi + P2Year^ + * Year, + 

Where 

i denotes a particular school; and 

t denotes the year that the survey measure belongs to (t=2005, 2007). 

Y„ is a survey measure of school i in year t, 

RF, is an indicator variable for RF school status. It is set to one if the school is a RF 
school and zero otherwise. 

Time, is an indicator variable for year 2007. It is set to one if t equals 2007 and zero 
otherwise. 

e„ : Usual error term. 

The coefficients in this model, alone and in combination, provide the estimates necessary for 
addressing the three research questions specified above. They are interpreted as follows: 

Status (Means) of RF or non-RF Title I schools in 2005 and 2007: 

Pq : Mean of the survey measure Y„ in non-RF Title I schools in 2005. 
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: Mean of the survey measure in RF schools in 2005. 

+ ^ 2 - Mean of the survey measure Y.^ in non-RF Title 1 schools in 2007. 

Mean of the survey measure Y.^ in RF schools in 2007. 

Longitudinal Change in RF and non-RF Title I Schools: 

P 2 ■ The change of Y.^ in non-RF Title I schools between 2005 and 2007. 

P 2 + Ps- The change of Y.^ in RF schools between 2005 and 2007. 



Differential Change Between the Two Groups of Schools: 

/?3 : This is the difference-in-differences estimate. It is equal to the difference 

between the change of in RF and non-RF Title I schools between 2005 
and 2007. 

Analysis of Student Reading Achievement 

Reading Achievement on State Reading Tests 

To investigate the relationship between Reading First and reading achievement, we analyzed 
grade-level scores on state reading assessments, as they appear in national databases for all 
schools nationwide. These analyses included reading scores from 12,362 schools, 3,000 RF and 
9,362 non-RF Title I schools. As described in Chapter 8, to conduct these analyses we had to be 
able to identify, for each school, those years (and their corresponding test scores) that occurred 
both before and after the implementation of RF. Because there is no one data source that tells us 
when each RF school received RF funds, we applied four different methods to define schools’ 
pre- and post years of RF implementation (state-level, district level using award date, district- 
level using an adjusted award date, and a school-level using a school award date). This means 
that we assigned dates for some RF schools (depending on the method) and all non-RF Title I 
schools. Below, we describe the procedures used to assign start dates for each of the four 
methods. 

Initial State Award Date Test scores in years prior to the initial state award date were designated 
as pre-RF implementation years. Scores after the last school in the state received funding are 
designated as post-RF implementation years. This allows sufficient time for the funds to have 
been distributed to all of the schools. Scores from the intervening years, when some schools 
may have been funded while others were not yet funded, are excluded from the analysis. In most 
cases, this strategy resulted in two years of data being excluded from the analyses. There was no 
imputation here for either RF or non-RF Title I schools because we had initial state award dates 
for all states and the dates were used to specify the onset of RF. All non-RF Title I schools were 
assigned the same initial state award date. This means that for this method all schools (both RF 
and non-RF Title I) in the analysis had the same initial start date and the same designations for 
pre- and post-RF implementation years. 



The last school funded was designated using from the Southwest Educational Development Laboratory (SEDL) 
Reading First Awards Database as of May 2005. 
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District Award Date District award dates provided by the U.S. Department of Edueation for all RE 
schools allowed us to identify RF schools’ pre- and post-RF implementation years. Non-RF 
Title I sehools in RF districts, about 40 percent of the non-RF Title I sehool sample, were given 
start dates proportional to the distribution of RF sehools’ award dates within eaeh RF distriet. 

For the non-RF Title I sehools in districts with no RF schools, 60 percent of the non-RF Title I 
school sample, start dates were imputed proportional to the distribution of RF sehools’ district 
award dates within each state. 

Adjusted District Award Date Analysis of 2005 survey responses from RF principals indicated 
that there is often a time lag between district- level and sehool-level reeeipt of RF funding. For 
this method, we used RF sehools’ survey data to compute average distriet- and state-level 
number of days to aeeount for the lag. The district award dates for all RF and non-RF Title I 
schools in RF districts were adjusted by adding the average distriet-level time lag. For all RF 
and non-RF Title I sehools not in RF districts were adjusted by adding the average state-level 
time lag. As was the ease for the analysis using the distriet award date, or the non-RF Title I 
schools in districts with no RF schools, start dates were imputed proportional to the distribution 
of RF sehools’ district award dates within each state. 



School Award Date Dates indicating when the sample of RF sehools received funding were 
provided by prineipals on their surveys. RF and non-RF Title I schools in RF districts that are 
represented in our study sample dates were assigned proportional to the school dates within 
those distriets. All other RF and non-RF schools, (both in RF and non-RF districts), we assigned 
the adjusted award date described above for RF schools. As a result, for this analysis, school 
award dates were imputed for all schools not in RF sample distriets, about 75 percent of RF 
schools and 70 percent of non-RF Title I sehools. 

For eaeh state, we eonducted analyses when we had at least one year of pre- and one year of 
post-RF implementation third- or fourth-grade test scores. Because we used multiple eriteria to 
define pre- and post-implementation years, in some states we eould not eonduct all four analyses. 
Third-grade analyses were conducted in 24 states; in 19 we condueted all four analyses. In five 
states, we condueted only the district and school-level analyses beeause the state pre-post 
strategy left these states without both a pre- and post-implementation year with test seores. 
Analyses of fourth-grade seores were eondueted in 17 states; in 10 we condueted four analyses, 
and in seven we eondueted only the district and school-level analyses. 

Fitting Statisticai Modeis 

For eaeh state, we applied one of two related estimation teehniques to model the statistical 
relationship between RF and student reading achievement: 1) a linear model, or 2) a baseline- 
mean model, both of which are deseribed below. 

Linear Modei. We fit statistieal models that compared RF sehools’ expected reading aehievement 
scores to their actual scores. These models also estimated pre- and post-RF achievement in non- 
RF Title I schools to control for any secular changes that may have occurred in the state. In 
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addition, the number of months between the award date specified in the model and the test score 
is included as a time-varying predictor at level one to control for differences in the amount of 
time schools could have been implementing the program prior to students being tested. 

To ensure that our analyses produce stable estimates, we imposed several conditions on using the 
linear model. First, in order to generate stable estimates of schools’ reading achievement prior to 
the implementation of Reading First, we required five or more years of pre-Reading First 
reading scores and two or more years of post-Reading First reading scores. Second, the 
trajectory of reading scores must be more accurately estimated using a linear model than by 
averaging the annual scores to estimate a mean.’^ At third grade, nine states met both of these 
conditions (Colorado, Hawaii, Illinois, Indiana, Kentucky, North Carolina, Oregon, South 
Carolina, and Virginia). At fourth grade, across all states, the data available could only support 
fitting baseline-mean models. 

The following two-level model was used to compare RF and non-RF Title I schools’ expected 
reading achievement scores to their actual scores: 

Level one (time): 

Yy = + K^jTIMEy + K^yPOSTy + K^ytnonthSy + Sy 

Level two (school): 

^2j=YiG +YnRFj +C 2 U 

where: 

Yy is the reading achievement score for school j at time /, 

Tiyj is the predicted initial (e.g., when time is 0) baseline reading achievement for school j, 

TIMEfj is a counter for time, for school j at time i, as it increases by 1 for each school year in 
which we have test score data, 

POST^j is an indicator that defines, for school j at time i, a time period prior to a state’s having 

received RF funds (POST=0), or a time period after a state has received RF funds 
(POST=l), 

month^j is a time-varying predictor indicating the number of months between the award date 
and test year i in school j, 



More specifically, the relationship between reading scores and time (year) would have to have a slope greater 
than zero. 

For Illinois, linear models were fit for all but the state model where a baseline mean model was used due to 
insufficient data to fit a linear model. 
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jc^j is the baseline linear slope describing changes in student achievement in school j over the 
baseline years, 

rc^j is the mean difference in school j ‘s reading scores pre- and post-RF implementation, and 

G 

is a random error term for school j at time i. 

is the mean initial baseline reading achievement across schools, 

RF 

^ is an indicator that for school j equals 1 in RF schools and 0 in non-RF Title I schools, 

is the difference in initial baseline achievement between RF and non-RF Title I schools, 
/jQ is the average slope describing linear change over time in student achievement in non-RF 
Title I schools, 

/jj is the difference in slope describing linear change over time between RF and non-RF 
Title I schools, 

/21 is the difference in difference between expected and actual scores between RF and non-RF 
Title I schools, and 

and are random error terms for school J. 

In the level-one model, model, is an estimate describing whether each school’s average 

student achievement changes over time. In the level-two model, /21 provides an estimate of the 
extent to which these differences are different, on average, in RF schools and non-RF Title I 
schools, allowing us to determine if RF schools experienced a larger increase in student 
achievement than the non-RF Title I schools, after the implementation of RF. If this estimate is 
positive and statistically significant, we would have evidence that student achievement in 
Reading First schools is higher than would have been expected given RF prior achievement and 
current achievement in non-RF Title I schools. 

Baseline-mean Model. This model is appropriate when there are too few data points to estimate a 
line, which was the case for 28 states. Instead, for these 28 states, we estimated a baseline mean 
model that averages the pre-RF student achievement scores. This technique is a traditional 
“difference-in-difference” model; it allows us to describe and compare the average performance 

O 1 

of RF schools and non-RF Title I schools both prior and subsequent to RF implementation. 
Again, the number of months between the award date specified in the model and the test score is 
included as a time-varying predictor at level one to control for differences in the amount of time 
schools could have been implementing the program prior to students being tested. 

It is important to note that the models we fit to estimate changes in student reading achievement 
deliberately have no additional covariates because these are purely descriptive analyses. We do 
not assume that we have created two equivalent groups of schools, either by design or by 
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The difference in difference approach estimates the pre-to-post change in average reading achievement scores 
for both RF and non-RF Title I schools, and then compares the two change scores. 
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statistical adjustment with covariates. These descriptive analyses examine outcomes in Reading 
First schools in the context of the non-RF Title I schools they are not impact analyses from 

89 

which we would make causal attributions. 

The following two-level model used data in each state, first to estimate mean student 
achievement before the state received RF funds, for both RF schools and non-RF Title I schools, 
and then, to compare the mean student achievement of the two groups of schools after RF 
implementation: 

Level one (time): 

Yy = tZqj + n^jPOSTy + n^jnionthsy + Sy 

Level two (school): 

^ij=Yw+ynRPj+‘^ij 

where: 

Yy is the reading achievement score for school j at time /, 
is the mean baseline reading achievement for school j, 

POSfj is an indicator for school j that defines a time period prior to a state’s having received RF 
funds (POST=0), or as a time period after a state has received RF funds (POST=l), 

month^j is a time-varying predictor indicating the number of months between the award date 
and test year i in school j, 

is the difference between the baseline mean and the follow-up mean for school j, and 
Sy is a random error term for school j at time i, 

Yqq is the mean reading achievement before the state received RF funding {baseline) for non-RF 
Title I schools, 

RFj is an indicator for school j that equals 1 in RF schools and 0 in non-RF Title I schools, 

/oj is the difference in achievement between RF and non-RF Title I schools at baseline, 

/jo is the average difference between the baseline mean and the follow-up mean for non-RF 
Title I schools. 



It should be noted that omitting covariates from the statistical models that might be correlated with reading 
achievement could affect the interpretation of the differences in achievement gains of RF and non-RF Title I 
schools. 
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/jj is the difference in the difference between the baseline mean and the follow-up mean 
between RF and non-RF Title 1 schools, and 

and are random error terms for school j. 

In the level-one model, is an estimate of each school’s difference between its baseline mean 
achievement score and its follow-up mean achievement score. In the second equation of the 
level-two model, provides an estimate of the extent to which these differences are different, 
on average, in Reading First schools and non-RF Title I schools, allowing us to determine if RF 
schools experienced a larger increase in student achievement than the non-RF Title I schools, 
after the implementation of RF. If this estimate is positive and statistically significant, we would 
have evidence that student achievement in Reading First schools is higher than would have been 
expected given RF prior achievement and current achievement in non-RF Title I schools. 

Calculating Effect Sizes 

In order to draw conclusions about schools’ reading performance across states and to benefit 
from the statistical power gained by combining data, we conducted a pooled analysis (separately 
for third- and fourth-grade) across all states included in these analyses. Because states reported 
on their students’ reading performance using different metrics, we converted each state’s 
difference-in-difference estimate into an effect size by dividing it by the standard deviation of 
scores of the non-RF Title I schools during pre-RF implementation years. Next, the state effect 
sizes were weighted using the inverse of the variance of the effect size. In this way, effect sizes 
from states that provided more precise difference-in-difference estimates were weighted more 
heavily than effect sizes estimated with less precision. Finally, these weighted effect sizes were 
averaged to obtain a pooled estimate of the difference in pre- to post-RF implementation gains in 
reading performance between RF and non-RF Title I schools. 

Reading Achievement and the Implementation of Reading First Activities 

To investigate the relationship between student reading achievement and the implementation of 
RF activities, we rely on two data sources: 1) third-grade reading scores on state assessments (as 
used above); and 2) selected responses from the spring 2005 teacher survey administration, from 
which we constructed four composite variables to characterize key aspects of teachers’ reading 
programs (described below). 

Because state tests and metrics vary from state to state, we standardized the reading achievement 
scores as follows. We designated schools as either high- or low-performing on the basis of 
their reading scores relative to other RF and non-RF Title I schools in their state. For these 
analyses, within each state, we defined high-performing schools as those with average 

Using the standard deviation from the ‘pre’ scores of non-RF Title I schools ensures that it is not affected by the 
Reading First program. 

We cannot conduct state-by-state analyses on these data; our school samples are nationally representative, 
however, they are not representative of individual states. 

C-8 Appendix C 





achievement scores in the top quartile, and low-performing schools as those with average scores 
in the lowest quartile.^^ We excluded schools that had reading scores in the middle 50 percent of 
the distribution resulting in a sample of 390 schools from 21 states. 

Fitting a Statisticai Modei 

We fit a logistic regression model using the four composites to predict the probability that a 
school scored in the top quartile (of all RF and non-RF Title I schools) on their state’s third- 
grade reading assessment. The model also included a) 20 dichotomous state indicator variables 
in order to account for state level variation in the probability of a school’s being in the top 
quartile on the reading assessment; and b) an indicator for RF or non-RF Title I status. 

1 + exp 

where: 

HQj is the highest quartile on a state’s third-grade state reading assessment; 1 indicates that a 
school scored in the top quartile (relative to other RF and non-RF Title I schools in their state), 



- 1 RF, + p, ST. + p, PD. + p, AS. + p, RI . + I p,^ SDV. 

s=\ 



RFj is an indicator for school j that equals 1 in RF schools and 0 in non-RF Title I schools, 

STj is the score on the composite based on activities for struggling readers for school j, 

PDj is the score on the composite based on participation in professional development activities 
for school j, 

AS J is the score on the composite based on teachers’ uses of assessment to inform their 
instruction for school j, 

RI J is the score on the composite based on activities for struggling readers for school j, and 
SDV^j is a series of 20 dichotomous state dummy variables for school j in s state s. 

The Composite Variables 

Selected responses from the spring 2005 teacher survey administration were used to construct 
four composite variables to characterize key aspects of teachers’ reading programs: 



Reading achievement levels are calculated using schools’ average scores in those states with more than one 
post-RF score because the average represents a more stable estimate of schools’ true average student 
achievement than would scores only for the most recent year available. 
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• Classroom reading instruction: the amount of time for reading instruction, alignment of 
instruction with scientifically based reading research (SBRR), and the types of direct 
support teachers receive for their reading instruction. 

• Help for struggling readers: placement of struggling readers in intervention programs, 
provision of extra instructional time; and use of materials that supplement the core 
reading program. 

• Professional development: the amount of hours teachers attended professional 
development activities focused on reading: professional development in (a) the five 
dimensions of reading instruction; (b) assisting in the instruction provided to struggling 
readers; and (c) using assessments to inform instruction; (3) teachers’ perceived level of 
preparedness to teach the five dimensions of reading. 

• Assessment: the use of test results to organize instructional groups or to determine 
students’ progress on skills: use of diagnostic tests to identify students who need reading 
intervention services; and use of screening tests to identify students who need a 
supplementary reading program. 

Exhibit C. 1 presents the specific variables included in each of the four composites. 

Each composite includes variables based upon different metrics. For example, the classroom 
reading instruction composite includes one item based upon minutes of reading instruction, and 
another based upon the percentage SBRR activities teachers rated as ‘central to their instruction.’ 
One commonly used strategy for interpreting and reporting on composite measures is to 
standardize the scores of the component items, so that the results can be presented on a common 
metric. For the composites described above, teachers’ responses to individual items were 
converted to z-scores (with a mean of 0 and a standard deviation of 1), and then averaged to 
generate a single score for each composite. The teacher-level scores were then aggregated to 
school-level z-scores for each composite. The use of z-scores enhances our capacity to 
characterize the results. For example, a one-point increase on the composite score is equivalent 
to an increase of one standard deviation, allowing us to describe a given score in terms of its 
distance from the mean score. 
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Exhibit C.1 



Composite Descriptions 



Teacher Survey Variables 


Survey 

Question 


Description 


Reading Instruction 


Instructional Time-length of reading 
block 


QC1 


Teacher average across five days. This average was then 
standardized'^ 


Alignment of instruction with SBRR 


QC6 (various 
items) 


Teacher scores computed based on the percent of RF- 
aligned activities teachers indicated were central to their 
instruction (scale=1 to 100%); scores were then 
standardized. 


Teachers’ receipt of direct support 
for reading instruction 


QD2a, d, h, I, j 


Teacher scores computed based on the percent of the 
types of direct assistance they received (scale=1 to 
100%); scores were then standardized. 



Struggling Readers 



Placement of struggling readers in 
intervention programs 


QC9j 


Teacher response to whether or not such support was 
received. 


Extra instructional time for struggling 
readers 


QC9f 


Teacher response to whether or not such support was 
received. 


Struggling readers-placement in 
materials that supplement core 
reading programs 


QC9g 


Teacher response to whether or not such support was 
received. 



Professional Development 



Number of hours attended PD in 
reading 


QD1 


Teacher responses were standardized. 


PD in a) five dimensions of reading 
instruction, b) assisting in instructing 
struggling readers, c) using 
assessments to inform instruction 


QD4 (various 
items); QD5j, k, 1, 
m, n, 0 , q 


Teacher scores based on the percent of topic items in 
which they received PD (Scale= 1 to 100%); scores were 
then standardized. 


Teacher Survey Variables 


Survey 

Question 


Description 


Perceived level of preparedness to 
teach five dimensions 


QD6 


An average level of preparedness across the five 
dimensions was computed (Scale=1 to 5); scores were 
then standardized. 



Assessment 



Use of test results to organize 
instructional groups 


QC6U 


Teacher response to whether or not teachers engaged in 
this activity. 


Use of test results to determine 
progress on skills 


QC6W 


Teacher response to whether or not teachers engaged in 
this activity. 


Use of diagnostic tests to identify 
students who need reading 
intervention services 


QC6Y 


Teacher response to whether or not teachers engaged in 
this activity. 



For each teacher, we computed the percent of these three activities in which they engaged. Teacher scores 
were then averaged within school to yield a school-level composite score on teacher use of assessment. 



To standardize we subtraeted a teaeher’s value from the mean value (based on our analytie sample) and divided by the standard 
deviation for that variable, thereby eonverting the seores to z-seores. 



The composites are more efficient at representing multiple individual variables potentially 
related to student reading achievement than are the individual variables themselves. Further, as 
described in below, analyses of the individual and composite variables indicate that the each of 
the composites accurately represent their component variables (Exhibit C.2). 
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Exhibit C.2 

Using Composite Components to Predict the Probabiiity of Schoois Scoring in the Top Quartiie 
on States’ Reading Tests 



Variable 


Estimate 


Marginal 

Effect 


Standard 

Error 


P-value 


struggling Readers-Composite 


.691 


.156 


.162 


.001 


Placement of struggling readers in intervention 
program 


.190 


.010 


.033 


.002 


Extra instructional time for struggling readers 


.166 


.087 


.033 


.007 


struggling readers-placement in materials that 
supplement core reading program 


.193 


.101 


.030 


.001 


Uses of Assessment-Composite 


-.091 


-021 


.167 


.587 


Use of test results to organize instructional 
groups 


-.011 


-.005 


.028 


.846 


Use of test results to determine progress on skills 


.006 


.003 


.029 


.904 


Use of diagnostic tests to identify students who 
need reading intervention services 


.003 


.002 


.028 


.953 


Reading Instruction -Composite 


-.268 


-.061 


.210 


.202 


Instructional Time-length of reading block 


.021 


.011 


.034 


.746 


Alignment of instruction with SBRR 


.004 


.002 


.032 


.946 


Teachers’ receipt of direct support for reading 
instruction 


-.086 


-.045 


.031 


.148 


Professional Development -Composite 


.180 


.040 


.167 


.278 


Number of hours attended PD in reading 


.063 


.034 


.033 


.310 


PD in phonemic awareness 


.045 


.024 


.037 


.511 


PD in decoding 


.052 


.027 


.036 


.448 


PD in vocabulary 


.098 


.052 


.031 


.095 


PD in fluency 


.054 


.029 


.033 


.381 


PD in comprehension 


-.003 


-.002 


.029 


.952 


PD in using assessments to inform instruction 


.020 


.010 


.032 


.750 


PD in assisting in instructing struggling readers 


.003 


.001 


.031 


.962 


Teachers’ perceived level of preparedness to 
teach the five dimensions of reading 


.099 


.052 


.029 


.077 
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Abt ID# // Barcode 
here.... 



1 - 6 / 

BATCH: 7-9/ 



0MB Number: 1875-0232 

Expiration Date: 10/31/07 



Reading First Impiementation Study 
Grade 1 Teacher Survey 



The U.S. Department of Edueation’s Policy and Program Studies Service has contracted with Abt 
Assoeiates Inc. to conduct a national evaluation of K-3 reading instruetion in Reading First and 
Title I sehools. The study’s data eollection ineludes two rounds of survey administration (2005 
and 2007) from both Reading First schools and Title I schools. Survey results from the 2005 
administration involved over 9,000 school-based respondents, and they have been summarized in 
a recent report. The Reading First Evaluation: Interim Report (July 2006). 

In each survey administration, we ask the prineipal, reading eoach, and a sample of K-3 teaehers 
from eaeh partieipating school to complete a questionnaire. Partieipants will help inform the U.S. 
Department of Edueation, Congress, polieymakers, practitioners, and researehers about how K-3 
reading instruetion is implemented in sehools and what strategies teaehers use to provide high- 
quality, evidenee-based reading instruction in grades K-3. 
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Additional Information 



The survey will take you approximately 30 minutes to complete. All responses to the survey will be 
kept confidential. All individual identifying information will be used only by persons on the research 
team. Information such as school location (state), participants’ general job titles, grades they teach, and 
gender will be included in the study data files to be submitted to the Department of Education. 
However, participants’ names will be stripped from all analysis data files and data files to be submitted 
to the Department of Education. We will not report any data about individual classrooms — all 
information will be reported at the grade and school levels. Neither your school nor your district will 
have access to any of the completed surveys at any time. 

Thank you for your cooperation with this survey! 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless such a collection displays a valid OMB control number. The valid 0MB control 
number for this information collection is 1875-0232. The time required to complete this information 
collection is estimated to average 30 minutes per response, including the time to review instruction, 
search existing data resources, gather the data needed, and complete and review the information 
collection. If you have any comments concerning the accuracy of the time estimate or suggestions for 
improving this form, please write to: Policy and Program Studies Service, U.S. Department of 
Education, 400 Maryland Avenue, SW, Washington, DC, 20202. 



Instructions 



Unless otherwise noted, your responses should reflect your experiences during the 2006-2007 
school year in the school to which this survey was sent. 



■ Please complete all questions; each question includes directions for recording your answer. 



■ You are sometimes told to skip over some questions in the survey. When this happens, you will 
see an arrow with a note that tells you what question to answer like this: 

□ i Yes 

□2 No ^ Skip to E4 



■ If you have any questions about how to complete the survey, please call: 1-xxx-xxx-xxxx. This 
is a free call and will connect you with our expert interviewers who can assist you. 
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A. Your Background and Experience 



A 1 . Including this year, how long have you been a teacher? If less than one year, please enter ‘ 1 ’ 





Enter # of years below. . . 


a. Total number of years as a teacher 


Years 

10 -11/ 


b. Number of years teaching at this school 


Years 

12 - 13 / 



A 2 . What grade(s) are you currently teaching? (Check all that apply) 



□ 1 Kindergarten iv 

□2 First grade 15/ 

□3 Second grade 16/ 

□4 Third grade 17/ 

□95 Other (Please specify): 18-19/ 20-21/ 



22-23/ 24-25/ 



A 3 . How well do you feel your pre-service teacher training prepared you to teach each of the 
following dimensions of reading? 



Pre-service teacher training refers to training you received before you became certified and began 
teaching. For those who began their teacher career through an alternative certification or emergency 
certification program, and began teaching before they were certified, pre-service teacher training refers 
to the training you received to become fully certified. 



Please ehoose a ‘ 1’ if you were ‘nc 
well prepared.’ 


)t at all prepared’ to teaeh the dimension and a ‘5’ if you were 

Check only one box for each item. . . 


‘extremely 


Not at all 
prepared 




Somewhat 

well 

prepared 


Extremely 

well 

prepared 


a. Phonemic awareness 


□i 


□2 


□3 °4 


□5 26/ 


b. Decoding 




^2 


□3 


□5 27/ 


c. Vocabulary 


^1 


^2 


□ 3 


□5 28/ 


d. Comprehension 






□3 


□5 29/ 


e. Fluency building 


□i 


□2 


□3 °4 


□5 30/ 
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B. 



Instruction and Assessment in Reading 



B 1 . Last week, approximately how many minutes per day did you devote to reading instruction? 

Include only reading instruction and not other language arts such as writing, spelling. Fill in the 
chart for each day last week with your best estimate of the number of minutes. . . 



Monday 


Tuesday 


Wednesday 


Thursday 


Friday 


# min. / day 


# min. / day 


# min. / day 


# min. / day 


# min. / day 


31-33/ 


34-36/ 


37-39/ 


40-42/ 


43-45/ 



B2. Has the average number of minutes you spend each day this year teaching reading decreased, 
remained the same, or increased from last year (2005-2006)? (Please check one) 

□ 1 I did not teach reading last year 46/ 

□2 Decreased 
□3 Remained the same 
□4 Increased 

B3. How often during this school year is time regularly scheduled and formally set aside during the 
school day for Grade 1 teachers to: 





Check only one box for each item ... 




Not at 
all 


1-4 

times 


5-8 

times 


Once a 
month 


Once a 
week or 
more 


Occurs only 
informally, 
as needed 


a. Collaborate on reading lesson planning and instruction. 


□i 


□2 


□3 


□4 


□5 


□ e 47 / 


b. Observe reading instruction in other classrooms. 




^2 


□3 


^4 




00 

□ 


c. Use assessment data to plan instruction. 


□i 


□2 




□4 


□5 


49/ 


d. Participate in coaching with or be coached about 
reading by other teachers. 


□i 


□2 


□3 


□4 


□5 


0 

□ 


e. Be coached about my reading instruction by a reading 
coach (see below). 




^2 


□ 3 


^4 




1^6 51/ 


f Coordinate reading interventions for struggling readers 
with special education staff. 


□i 


□2 


□3 


□4 


□5 


□ e 52/ 


g. Coordinate reading interventions for struggling readers 
with English language learner (ELL) staff (see below). 




^2 


□ 3 


^4 


^5 


□5 53/ 


^ ^ Check if no ELLs H ^ 


54/ 



A reading coach is a staff member whose primary role is to provide ongoing training and support to classroom teachers in 

the delivery of effective reading instruction. This assistance may include planning instruction, providing demonstration lessons, 
observing and providing feedback, using assessment results to guide instruction, etc. 



English language learner (ELL) indicates a student who is in the process of acquiring English and has a first language other 
than English. Other common related terms include language minority or limited English proficient (LEP) students, students in 
English as a second language (ESL), or students in classes for English for speakers of other languages (ESOL). 
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B4. Please deseribe your use of the following reading instmetional aetivities this year. 

• Check column A ONLY if the instructional activity is one that you use frequently when you teach reading or 
one on which you rely heavily in your reading instruction. 

• Check column B if you use the instructional activity, but it is a small part of your teaching, and not one you use 
frequently. It might be an activity that you use if there is time, but it is not one on which you rely heavily for 
your reading instruction. 

• Check column C if the activity is not one you use in your reading instruction. 







Check only one box for each item... 






A 

Central to mv 
reading 
instruction 


B 

Small part of 
my reading 
instruction 


c 

Not Part of my 
reading 
instruction 




a. I provide feedback on errors as students read orally. 


□i 


□2 


□3 


68/ 




b. Students read texts that are easy to decode. 


n. 




□3 


55/ 




c. Students read silently. 


n. 




□3 


60/ 




d. Students reread familiar stories. 


□i 


□2 


□3 


58/ 


Reading 


e. Students select books from the library for 
independent reading. 


□i 


□2 


□3 


117/ 


text 


f I develop language experience stories with my class. 


n. 


^2 


□3 


70/ 




g. Pairs of students read aloud together. 


n. 


^2 


□3 


71/ 




h. Students read aloud with expression and proper 
phrasing. 


□i 


□2 


□3 


72/ 




i. Students reread to find facts to answer questions. 


□i 


□2 


□3 


73/ 




j. Class creates story maps. 




^2 


□3 


74/ 




k. I listen to students read aloud without correcting 
errors. 


□i 


□2 


□3 


61/ 




1. Students isolate sounds in words that I say. 


□i 


□2 


□3 


94/ 




m. Students practice naming letters. 


n. 


^2 


□3 


95/ 




n. Students blend phonemes to form words. 


n. 


^2 


□3 


96/ 


Work 

with 

sounds 


0 . Students practice reading high frequency words for 
automaticity. 


□i 


□2 


□3 


88/ 


p. Students use knowledge of root words, prefixes, and 
suffixes to decode new words. 


□i 


□2 


□3 


89/ 


and 

words 


q. I stop students while reading and have them self- 
correct misidentified words. 


□i 


□2 


□3 


91/ 


r. Students use pictures to identify unknown words. 


□i 


□2 


□3 


97/ 




s. I teach decoding skills while reading stories. 


n. 


^2 


□3 


85/ 




t. Students practice writing words as separate syllables. 


n. 


°2 


□3 


93/ 




u. I teach decoding skills with word families. 


□i 


□2 


□3 


98/ 



56/BLANK 

57/BLANK 

59/BLANK 

62-67/BLANK 

69/BLANK 

75-83/BLANK 



84/BLANK 

86/BLANK 

87/BLANK 

90/BLANK 

92/BLANK 

99-107/BLANK 
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B4. CONTINUED. Please deseribe your use of the following reading instmetional aetivities this year. 

• Check column A ONLY if the instructional activity is one that you use frequently when you teach reading or 
one on which you rely heavily in your reading instruction. 

• Check column B if you use the instructional activity, but it is a small part of your teaching, and not one you use 
frequently. It might be an activity that you use if there is time, but it is not one on which you rely heavily for 
your reading instruction. 

• Check column C if the activity is not one you use in your reading instruction. 





Check only one box for each item... 




A 

Central to mv 
reading 
instruction 


B 

Small part of 
my reading 
instruction 


c 

Not Part of my 
reading 
instruction 




V. I engage students in rhyming games and songs. 


□i 


□2 




120/ 




w. Students retell stories in sequence and identify 
characters and main events. 


□i 


□2 


□3 


121/ 




X. I read stories aloud to students. 






□ 3 


122/ 




y. Students write stories using invented spelling. 


□i 


□2 


□3 


123/ 




z. I discuss new and unusual words before reading. 


□i 


□2 


□3 


112/ 


other 

Techniques 


aa. Students write vocabulary words in sentences. 






□ 3 


110/ 


bb. Students read stories they have written to others. 


□i 


□2 


□3 


115/ 




cc. Students make predictions while reading stories. 


□i 


□2 


□3 


124/ 




dd. Students use dictionaries to find word meanings. 




^2 


□ 3 


111/ 




ee. Students are given time to read on their own for 
enjoyment. 


□i 


□2 


□3 


118/ 




ff Students develop questions about text material. 


□i 


□2 


□3 


125/ 




gg. Students act out story as a play. 




^2 


□ 3 


126/ 



108/BLANK 

109/BLANK 

113/BLANK 

114/BLANK 

116/BLANK 

119/BLANK 

127-134/BLANK 
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B5. Please describe your use of the following teaching strategies and materials this year. 

• Check column A ONLY if the item is one that you use frequently or one on which you rely heavily in your reading 
instruction. 

• Check column B if you use the item, but it is a small part of your teaching, and not one you use frequently. It may be 
an approach you use if there is time, but it is not one on which you rely heavily. 



• Check column C if the item is not one you use in your reading instruction. 





Check only one box for each item... 


A 

Central to 
my reading 
instruction 


B 

Small part of 
my reading 
instruction 


c 

Not Part of 
my reading 
instruction 




a. Provide time in reading block for skill practice on own. 






□3 


135/ 




b. Provide materials for at-home practice of skills 


n, 


n. 


n. 


136 / 




introduced in class. 




2 


3 






c. Provide extra reading instructional time for struggling 


n, 








Instruction 


students. 




‘—‘2 


‘—‘3 






d. Include writing opportunities in reading instruction. 




□2 


□3 


138/ 




e. Build spelling practice into reading instruction. 




□2 


□3 


139/ 




f Develop reading skills through science and social studies. 






□3 


140/ 




g. Teach whole class reading lessons. 




^2 


□3 


141/ 




h. Work one-to-one with students on reading. 




□2 


□3 


142/ 




i. Work with small groups of students. 




□2 


□3 


143/ 


Grouping 


j. Group students based on skill levels. 




^2 


□3 


144/ 




k. Group students based on mixed abilities (cooperative 


n, 










groups). 




‘—‘2 


‘—‘3 






1. Pair strong readers with those with weaker skills. 




□2 


□3 


146/ 




m. Use core reading series. 




^2 


□3 


147/ 




n. Use supplementary reading materials for instruction in 












the following areas: 












1 . Phonemic awareness 




^2 


□3 


148/ 




2. Phonics 




□2 


□3 


149/ 




3. Fluency 




^2 


□3 


150/ 




4. Vocabulary 




°2 


□3 


151/ 


13 aI I n A1 












fxSoQiny 

materials 


5. Comprehension 




□2 


□3 


152/ 




0 . Use children’s trade books. 




^2 


□3 


153/ 




p. Use books that are easy to decode. 




^2 


□3 


154/ 




q. Use books with patterned predictable language. 


□i 


□2 


□3 


155/ 




r. Use separate intervention materials for some students. 




^2 


□3 


156/ 




s. Use reading software/technology. 




^2 


□3 


157/ 




t. Use teacher-made materials. 




□2 


□3 


158/ 



Supplementary Reading Materials provide additional instruction in a targeted area of reading to all students. Do not include 
materials that are used only with struggling readers. Include teacher-made materials, if applicable. 
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B5. CONTINUED. Please deseribe your use of the following teaehing strategies and materials this year. 

• Check column A ONLY if the item is one that you use frequently or one on which you rely heavily in your reading 
instruction. 

• Check column B if you use the item, but it is a small part of your teaching, and not one you use frequently. It may 

be an approach you use if there is time, but it is not one on which you rely heavily. 

• Check column C if the item is not one you use in your reading instruction. 







Check only one box for each item... 






A 

Central to 
my reading 
instruction 


B 

Small part of 
my reading 
instruction 


c 

Not Part of 
my reading 
instruction 




U. 


Use test results to organize instructional groups. 






□3 159/ 




V. 


Use informal reading inventories. 






□3 160/ 




W. 


Use tests to determine progress on skills. 


□i 


□2 


1^3 161/ 


Assessments 


X. 


Use tests to determine who can benefit from the core 
reading series. 




^2 


□3 162/ 




y- 


Use diagnostic tests to identify students who need 
reading intervention services. 




^2 


□3 163/ 




z. 


Use screening tests to identify students who need a 
supplementary reading program. 




^2 


□3 164/ 




aa. 


Conduct miscue analysis, analyzing errors students 
make while reading aloud. 




□2 


1^3 165/ 
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B6. What specific formal or informal assessments do you primarily use for placing and /or grouping 
students, determining student mastery of skills, and identifying core deficits of struggling students? 
If you use more than one assessment, please report only on the one that you use the most often. 
Please be as specific as possible when naming or describing the assessment(s). 

• In column A enter the name of the primary assessment used for each purpose. 

• In column B check the number of times the assessment is given during the school year. 

• In column C check whether students are usually assessed individually, in small groups, or in a whole class. 

• In column D check the average time that it takes to administer the assessment. 

• In column E check whether the assessment is used for accountability purposes for the Reading First program. No 
Child Left Behind (NCLB), or another program. Please check all that apply in this column only. 





Check only one box per column for each 
item... 


Check all that 
apply... 


A. 

Primary purposes and names of assessments 


B. C. D. 

Number of Students are Average time 

times given per assessed... it takes to 

school year administer 

assessment 

(Check one) (Check one) (Check one) 


E. 

Accountability 

purposes 

(Check all that 
apply) 


a. Placement and/or grouping students 

(Check one): 

166/ 

1 □ Assessment: 


iD 1 iD Individually iD 1-15 

2 ^ 2 2 ^ In small minutes 

sD 3 groups 2 ^ 16-30 

4^4 3 D In whole sD 31-45 

5 ^ 5 or more class 4 ^ 46-60 

5 ^ 61 or 
more 

173/ 174/ 175/ 


iD Reading 
First 

176/ 

2 U NCLB 

177/ 

3 ^ Other 

178/ 


167-168/ 

169-170/ 

171-172/ 

2 ^ I do not use an assessment for this purpose 


b. Determining student mastery of skills 

(Check one): 

179/ 

1 □ Assessment: 


iD 1 iD Individually iD 1-15 

2 ^ 2 2 ^ In small minutes 

sD 3 groups 2 ^ 16-30 

4^4 3 D In whole 3 ^ 31-45 

sD 5 or more class 4 ^ 46-60 

5 ^ 61 or 
more 

186/ 187/ 188/ 


iD Reading 
First 

189/ 

2 U NCLB 

190/ 

3 ^ Other 

191/ 


180-181/ 

182-183/ 

184-185/ 

2 ^ I do not use an assessment for this purpose 


c. Identifying the core deficits of struggling 
students (Check one): 

192/ 

1 □ Assessment: 


iD 1 iD Individually iD 1-15 

2 ^ 2 2 ^ In small minutes 

3 ^ 3 groups 2 ^ 16-30 

4^4 3 ^ In whole 3 ^ 31-45 

5 ^ 5 or more class 4 ^ 46-60 

5 ^ 61 or 
more 

199/ 200/ 201/ 


iD Reading 
First 

202/ 

2 U NCLB 

203/ 

3 ^ Other 

204/ 


193-194/ 

195-196/ 

197-198/ 

2 ^ I do not use an assessment for this purpose 
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B7. What materials are used with English Language Learners (ELLs) to whom you teach reading 



(see definition of ELL on page 2)? 

(Cheek all that apply) 

□ 1 Do not teach ELLs ^Skip to B8 205/ 

□2 Core reading program materials in the native language of the ELL 206/ 

□3 ELL students use the same materials as other students 207/ 

□4 Core reading program materials, plus supplementary/intervention resourees written in 

the eel’s native language 208/ 

□5 Core reading program materials, plus supplementary/intervention resourees written in 

English especially for ELLs 209/ 

□e Alternative core reading program materials in English geared toward the instructional 

level of the ELL 210 / 
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B8. What additional supports have students who are struggling readers received in the last month? 
Check whether or not your students who are struggling readers received each of the supports 



during the past month. 


Check only one box for each item... | 


Supports for Struggling Readers 


Received 


Did not receive 


a. 


Diagnostic assessment to determine core deficits. 




^2 


211/ 


b. 


Extra practice in the classroom with phonemic awareness. 






212/ 


c. 


Extra practice in the classroom with decoding. 




□2 


213/ 


d. 


Extra practice in the classroom with fluency. 




□2 


214/ 


e. 


Extra practice in the classroom with comprehension. 




^2 


215/ 


f. 


Extra instructional time. 




□2 


216/ 


g- 


Placement in materials that supplement the core reading 
program. 




□2 


217/ 


h. 


Placement in different level of core reading program. 




^2 


218/ 


i. Placement in separate core reading program. 




^2 


219/ 


j- 


Placement in special intervention program. 




□2 


220/ 


k. 


Work with tutor on one-to-one basis. 




^2 


221/ 


1. 


Work with reading specialist on one-to-one basis. 




^2 


222/ 


m. 


Work with reading specialist in small group. 




□2 


223/ 


n. 


Work with more advanced peer. 




^2 


224/ 


0. 


Special materials for parents to provide practice. 




^2 


225/ 



B9. What additional supports have students who are struggling readers and ELLs received in the last 
month? Check whether or not your students who are struggling readers and ELLs received each 
of the supports during the past month (see definition of ELL on page 2). 

Check if no ELLs ► Di Skip to Cl 

226/ 









Check only one box for each item ... 


Supports for Struggling Readers who are ELLs 


Received 


Did not receive 


a. 


instruction. 


English as a Second Language 


□1 


n 2 227/ 


b. 


language. 


Provide reading instruction in home 


□1 


n 2 228/ 


c. 


teacher. 


In classroom help in reading from ELL 


□1 


n 2 229/ 
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c. 



Cl. 



Professional Development in Reading for K-3 Teachers 

During the current school year, including summer 2006, did you participate in any professional 
development activities in reading? 

230 / 



Yes 

No ^2 ► Skip to C2 

If yes, in how many of each of the following types of professional development activities in 
reading have you participated? Please count each activity only once. What is the total number 
of hours you spent in these activities? 

First, write in the number of activities of each type in which you have been engaged. Then, write 
the total number of hours you spent in these activities. Mark 0 if you participated in none. 



Enter # and hours below... 





# of Different 
workshops 


Total hours 


a. Attended short, stand-alone training or workshop in reading (half-day or 
less) 


# 


Hours 


231 - 232 / 


233 - 235 / 


b. Attended longer institute or workshop in reading (more than half-day) 


# 


Hours 




236 - 237 / 


238 - 240 / 


c. Attended a college course in reading (include any courses you are 
currently attending) 


# 


Hours 


241 - 242 / 


243 - 245 / 


d. Attended a conference about reading (might include multiple short 
offerings) 


# 


Hours 


246 - 247 / 


248 - 250 / 
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C2. Below is a list of professional development aetivities that are often used to provide ongoing, 
direet support to teaehers for teaehing reading. 

• In the first column, please indicate whether you have received any of the following types of assistance/support for 

teaching during the current school year, including summer 2006 . 

• If you did not receive that type of support, please indicate whether the support was available, but you did not 

receive it (column 2), or if it was not available at your school (column 3). 





Check only one box for each item ... 




Types of assistance 
I received this year 


Available, but I 
did not receive 


Not available at 
my school 


a. Coaching or mentoring by reading coach in programs, 
materials, or strategies. 


n. 


□2 




251/ 


b. Coaching or mentoring from fellow teacher. 


□i 


□2 


□3 


252/ 


c. Peer study group or collegial circle for group study. 


n. 




□ 3 


253/ 


d. Demonstrations in my classroom. 


n. 


□2 


□3 


254/ 


e. Observations of other teachers. 


□i 


□2 


□3 


255/ 


f Diagnostic testing help from a reading coach or specialist 
for individual students. 


□i 




□ 3 


256/ 


g. Intervention service help from a reading coach or specialist 
for individual students. 


n. 


□2 


°3 


257/ 


h. Interpretation of assessment data. 


n. 


^2 


□ 3 


258/ 


i. Grade level meetings devoted to reading. 


n. 


^2 


□ 3 


259/ 


j. Using assessment data to determine topics that require 
additional instruction or practice. 


n. 


□2 


□3 


260/ 



C3. During the current school year, including summer 2006, approximately how many of the 

reading professional development activities for K-3 teachers: (Please choose the category 
that most closely describes your professional development.) 







Check only one box for each item... 






None 


One- 


One- 


Three- 


All 








Quarter 


Half 


Quarters 




a. 


were also attended by the principal? 


□1 


□2 


□3 


□4 


□5 


261/ 


b. 


provide teachers options among which to choose? 




^2 


□3 






262/ 


c. 


provide a stipend? 


□1 


□2 


□3 


□4 


□5 


263/ 


d. 


provide follow-up activities? 


□1 


□2 


□3 


□4 


□5 


264/ 


e. 


include release time for participating teachers? 




^2 


□3 


^4 




265/ 


f 


offer graduate college credits? 


□1 


□2 


□3 


□4 


□5 


266/ 


g- 


are held in a convenient location (e.g., activities held at school)? 


□1 


□2 


□3 


□4 


□5 


267/ 


h. 


use a team-based approach (joint training of people who work 
together)? 




^2 


□3 


^4 




268/ 


i. 


are given by trainers or facilitators who have a well-established 
reputation? 




^2 


□3 


°4 


^5 


269/ 
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C4. Below is a list of topics that are often covered in professional development activities designed to 
provide teachers with new information about the content of reading instruction. 

• In column A . identify the topics that were addressed in professional development activities in which you participated 
during the current school year, including summer 2006 . 

• In column B . please identify the topics in which you need more professional development, whether or not this 
school’s professional development activities have covered these topics. 

• Please check all that apply in columns A and B. 



Professional development is defined as any activity in which a teacher has learned about reading or 
reading instruction. This includes school-based workshops, meetings with reading coaches, and 
meetings with a study group of other teachers. 







CHECK ALL THAT APPLY 






A. Topics 
addressed in 
professional 
development 


B. Topics in which 
I need more 
professional 
development 


Phonemic 

Awareness 


a. Building phonological awareness, e.g. rhymes, dividing 
spoken language into sentences, words, syllables 


n. 


270/ 


^2 


271/ 


b. Identifying, adding, deleting sounds in spoken words 


□ 3 


272/ 




273/ 




c. Blending phonemes to form words 


n. 


274/ 




275/ 




d. Teaching letter-sound correspondence 


□3 


276/ 


□4 


277/ 


Decoding 


e. Teaching letter patterns (blends, digraphs, diphthongs) 


□1 


278/ 


□2 


279/ 


f Using syllable patterns to read words 


□ 3 


280/ 


^4 


281/ 




g. Teaching component parts: roots, prefixes, suffixes 


n. 


282/ 


^2 


283/ 




h. Teaching use of dictionary, thesaurus 


□3 


284/ 


□4 


285/ 


Vocabulary 


i. Direct teaching of vocabulary words and their meaning 


□1 


286/ 


□2 


287/ 




j. Antonyms and synonyms 


□ 3 


288/ 


^4 


289/ 




k. Teaching sight words 


n. 


290/ 


^2 


291/ 


Fluency 


1. Guided oral reading 


□3 


292/ 


□4 


293/ 




m. Encouraging expression while reading 


□1 


294/ 


□2 


295/ 




n. Setting motivation/asking prediction/preview questions 


□3 


296/ 


^4 


297/ 




0 . Constructing information about character, setting, and main 
events 




298/ 


^2 


299/ 


Comprehension 


p. Summarizing main ideas in narrative and informational text 


□3 


300/ 


□4 


301/ 


q. Self-monitoring strategies 


□1 


302/ 


□2 


303/ 




r. Asking questions at different levels (literal, inferential) 


□3 


304/ 


^4 


305/ 




s. Strategies for organizing text structure, e.g. story maps 


□1 


306/ 


□2 


307/ 



308-309/BLANK 
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C5. Below is a list of topics that are often covered in professional development activities that are 

designed to provide teachers with new information about teaching strategies used during reading 
instruction. 

• In column A , identify the topics that were addressed in professional development activities in which you 
participated during the current school year, including summer 2006 . 

• In column B , please identify the topics in which you need more professional development, whether or not this 



school’s professional development activities have covered these topics. 
• Please check all that apply in columns A and B. 







CHECK ALL THAT APPLY 


Topic 




A. Topics 
addressed in 
professional 
development 


B. Topics in which I 
need more 
professional 
development 




a. How to use the core reading program 


310/ 


□2 


311/ 




b. How to use children’s literature to teach reading 


□3 

312/ 


□4 


313/ 




c. How to use reading research to guide content of instruction 


□1 

314/ 


□2 


315/ 


Teaching 

Strategies 


d. How the core reading program incorporates research principles 


□ 3 

316/ 




317/ 


e. How to use the supplemental reading program(s) 


□i 

318/ 


□2 


319/ 




f How to integrate reading and writing instruction 


□3 

320/ 


□4 


321/ 




g. Strategies for teaching reading to ELLs (See definition of ELL 
on page 2) 


□ 1 322/ 




323/ 


Grouping 


h. Learning styles 


□ 3 

324/ 


^4 


325/ 


i. How to organize small group instruction 


n, 

326/ 




327/ 




j. How to diagnose reading problems 


□3 

328/ 


□4 


329/ 


Assessment 


k. How to administer assessments 


□1 

330/ 


□2 


331/ 




1. How to interpret and use assessment data to guide instruction 


□ 3 

332/ 


^4 


333/ 




m. How to help struggling readers with decoding 


n, 

334/ 


^2 


335/ 


struggling 

Readers 


n. How to help struggling readers with vocabulary 
0 . How to help struggling readers with comprehension 


□3 

336/ 

□1 

338/ 


□4 

□2 


337/ 

339/ 




p. How to motivate readers 


□3 

340/ 


^4 


341/ 




q. Strategies for teaching reading to students with diagnosed 
learning disabilities 


n, 

342/ 


^2 


343/ 




r. How to use state/district content standards for curriculum 
planning and teaching 


□3 

344/ 


□4 


345/ 


Organization/ 


s. How to align reading curriculum and instruction with 
state/district assessments 


□1 

346/ 


□2 


347/ 


planning 


t. How to work with parents 


□3 

348/ 


^4 


349/ 




u. Classroom management 


n, 

350/ 


^2 


351/ 
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C6. How well do you feel the professional development activities in which you participated during 
the current school year (including summer, 2006) prepared you to teach each of the following 
dimensions of reading? Please choose a ‘1’ if you were ‘not at all prepared’ to teach the 
dimension and a ‘5’ if you were ‘extremely well prepared.’ 



Check only one box for each item ... 





Not at all 
prepared 




Somewhat 
well prepared 




Extremely 
well prepared 


a. Phonemic awareness 






□3 


^4 


n 5 352/ 


b. Decoding 




□2 




□4 


□5 353/ 


c. Vocabulary 




°2 


□ 3 


^4 


n 5 354/ 


d. Comprehension 






□ 3 


^4 


n 5 355/ 


e. Fluency building 




□2 




□4 


n 5 356/ 



D. Support for Teaching Reading 



Dl. The next set of statements is about your reading program. Please indicate the extent to which 
you agree or disagree with each statement. 



Check only one box for each item ... 





Strongly 

Agree 


Agree 


Disagree 


Strongly 

Disagree 


a. I feel I need to make changes in the methods I use to teach children 
to read. 






□3 


^4 


357/ 


b. Other faculty/staff members have helped me to understand the 
difficulties that some children have in learning to read. 


□1 


□2 


□3 


□4 


358/ 


c. I have benefited from opportunities to learn more about methods 
for teaching reading. 




^2 


□3 


^4 


359/ 


d. The children in my class are making satisfactory progress in 
learning to read. 


□1 


□2 


□3 


□4 


360/ 


e. I do not have sufficient materials to teach reading effectively. 




^2 


□3 


^4 


361/ 


f I do not understand why some children learn to read easily while 
other children struggle to learn basic reading skills. 


□1 


□2 


□3 


□4 


362/ 


g. The reading coach supports my efforts to teach reading effectively. 




^2 


□3 


^4 


363/ 


h. I have a good understanding of how children acquire language and 
literacy skills. 


□1 


□2 


□3 


□4 


364/ 


i. I wish I had more opportunities to discuss how to teach reading 
with other teachers. 




^2 


□3 


^4 


365/ 


j. I know the current reading skill levels of all my students. 


□1 


□2 


□3 


□4 


366/ 


k. I know how to assess the progress of my students in reading. 




^2 


□3 


^4 


367/ 


1 . I have changed my methods of teaching reading as a result of 
professional development in reading. 


□1 


□2 


□3 


□4 


368/ 



Thank you for your cooperation and for taking time to answer these questions. Please place the 
completed survey in the enclosed envelope, seal the envelope and return it to your evaluation liaison. 

If you have any questions about the survey, please call 1-xxx-xxx-xxxx. 



D-18 



Survey of Grade 1 Teachers 




This is a placeholder for your back cover of the booklet - it should be blank! 
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Abt ID / barcode here 


1-6/ 


0MB Number: 1875-0232 


Rf version 


BATCH: 7-9/ 


Expiration Date : 10/31/07 



Reading First Impiementation Study 
Principal Survey 



The U.S. Department of Edueation’s Policy and Program Studies Service has contracted with Abt 
Associates Inc. to conduct a national evaluation of K-3 reading instruction in Reading First and Title I 
schools. The study’s data collection includes two rounds of survey administration (2005 and 2007) from 
both Reading First schools and Title I schools. Survey results from the 2005 administration involved 
over 9,000 school-based respondents, and they have been summarized in a recent report. The Reading 
First Evaluation: Interim Report (July 2006). 

In each survey administration, we ask the principal, reading coach, and a sample of K-3 teachers from 
each participating school to complete a questionnaire. Participants will help inform the U.S. Department 
of Education, Congress, policymakers, practitioners, and researchers about how K-3 reading instruction 
is implemented in schools and what strategies teachers use to provide high-quality, evidence-based 
reading instruction in grades K-3. 
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Additional Information 



The survey will take you approximately 30 minutes to eomplete. All responses to the survey will be 
kept eonfidential. All individual identifying information will be used only by persons on the research 
team. Information such as school location (state), participants’ general job titles, grades they teach, and 
gender will be included in the study data files to be submitted to the Department of Education. 
However, participants’ names will be stripped from all analysis data files and data files to be submitted 
to the Department of Education. We will not report any data about individual classrooms — all 
information will be reported at the grade and school levels. Neither your school nor your district will 
have access to any of the completed surveys at any time. 

Thank you for your cooperation with this survey! 



According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless 
such a collection displays a valid OMB control number. The valid 0MB control number for this information collection is 
1875-0232. The time required to complete this information collection is estimated to average 30 minutes per response, 
including the time to review instruction, search existing data resources, and gather the data needed, and complete and review 
the information collection. If you have any comments concerning the accuracy of the time estimate or suggestions for 
improving this form, please write to; Policy and Program Studies Service, U.S. Department of Education, 400 Maryland 
Avenue, SW, Washington, DC, 20202. 

Instructions 

Unless otherwise noted, your responses should reflect your experiences during the 2006-2007 
school year in the school to which this survey was sent. 

■ Please complete all questions; each question includes directions for recording your answer. 

■ You are sometimes told to skip over some questions in the survey. When this happens, you will 
see an arrow with a note that tells you what question to answer like this: 

□ i Yes 

□2 No ^ Skip to E4 

■ If you have any questions about how to complete the survey, please call: x-xxx-xxx-xxxx. This 
is a free call and will connect you with our expert interviewers who can assist you. 
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A. Background Information on You and Your School 



Al. Including this year, how many years have you been at this school in this position? 

(If less than one year, please enter ‘1’) 


Years 10-11/ 


A2. Including this year, what is the total number of years you have served as a principal? 

(If less than one year, please enter ‘1’) 


Years 

12 - 

13/ 


A3. Please provide the following information about students in your school for the current 
year (2006-2007): 




a. Total number of students currently enrolled 


Students 

14-17/ 


b. Percentage of students who have left the school at any point during the year, including 
the summers between school years, excluding those who have left having completed the 
highest grade available at your school (i.e., mobility rate) 


% 18-20/ 


c. Average attendance rate 


% 

21 - 

23/ 


d. Percentage of students in your school who are English Language Learners (ELLs) 

1 ► 


% 

24- 

26/ 


Check if no ELLs 


□ 1 27/ 



English language learner (ELL) indicates a student who is in the process of acquiring English and has a first 
language other than English. Other common related terms include language minority or limited English proficient 
(LEP) students, students in English as a second language (ESL), or students in classes for English for speakers of 
other languages (ESOL). 



A 4 . Compared to 5 years ago, has student enrollment in your school decreased, remained stable, or 
increased? 28 / 

□ 1 Decreased 
□2 Remained stable 
□3 Increased 

□4 Not applicable (school is new) 

A 5 . Please indicate the approximate percentage of students in each grade who: 



Enter % below for each grade level ... 



In 2006-2007, the percentage of students who... 


K 


1 


2 


3 


a. participate in interventions for struggling readers 


% 


% 


% 


% 


29-31/ 


32-34/ 


35-37/ 


38-40/ 


b. receive special education services 


% 


% 


% 


% 


41-43/ 


44-46/ 


47-49/ 


50-52/ 


c. receive ESL instruction 


% 


% 


% 


% 


53-55/ 


56-58/ 


59-61/ 


62-64/ 


d. receive reading instruction in a language other 
than English 


% 


% 


% 


% 


65-67/ 


68-70/ 


71-73/ 


74-76/ 


e. read at or above grade level 


% 


% 


% 


% 


77-79/ 


80-82/ 


83-85/ 


86-88/ 
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A6. How many classroom teachers are assigned to grades K-3 this year (2006-07)? How many of 
them are considered to be highly qualified? Please inelude regular edueation elasses only. 

• In column A , enter the total number of teachers in each grade. 

• In column B , enter the percentage of those teachers listed in each grade who are highly qualified. 



Highly qualiHed teachers have full state certification, at least a bachelor’s degree, and proven knowledge in 
the subject that they teach. 



Grade Level 


A. 

Enter # of classroom teachers 
below: 


B. 

Enter % who are highly 
qualified below: 


K 


# 


89 - 90 / 


% 


91 - 93 / 


1 


# 


94 - 95 / 


% 


96 - 98 / 


2 


# 


99 - 100 / 


% 


101 - 103 / 


3 


# 


104 - 105 / 


% 


106 - 108 / 


Other K-3 teachers (e.g., 
combination classrooms): 


# 


109 - 110 / 


% 


111 - 113 / 



A7. How many special education teachers are assigned to students in grades K-3? 

Teachers 



114 - 115 / 



A8. How many teaehers who speeialize in working with ELLs or teaehing English for speakers of other 
languages (ESOL) are assigned to students in grades K-3 (see definition of ELL on page 1)? 



Teachers 



116 - 117 / 



^ Cheek if no ELLs in grades K-3. I^i 

118 / 



A9. How many of eaeh type of reading support personnel does your sehool have for grades K-3? 



Type 


Insert # below . . . 


a. Certified non-classroom teachers, including special education teachers. Title I 


# 


119 - 


teachers, and reading coaches or specialists 


120 / 




b. Instructional aides or assistants (during school day) 


# 


121 - 


c. Tutors (before or after school) 


122 / 

# 


123 - 




124 / 





A reading coach is a staff member whose primary role is to provide ongoing training and support 
to classroom teachers in the delivery of effective reading instruction. This assistance may include 
planning instruction, providing demonstration lessons, observing and providing feedback, using 
assessment results to guide instruction, etc. 
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AlO. 



How many certified staff positions has your sehool added this year (such as a reading eoaeh) to 
support reading instruction in Grades K-3? If no positions were added, write 0. 

125 - 126 / 

Number of staff positions added: # 



All. Did your sehool make “adequate yearly progress” in reading/language arts on the basis of 2005- 
06 test seores, according to NCLB accountability provisions? (Please check only one.) 



“Adequate yearly progress” (AYP) is the amount of yearly improvement each school is expected to make. 

Each state is responsible for defining AYP and for determining the methods used to measure AYP. 

127 / 

□ i Yes 

□2 No 

1^3 Not sure/don’t know 
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B. Resources and Support for Your School’s Reading Program 



Bl. What sources of funding are being or have been used to support your school’s reading program 
this year (2006-2007)? (Check all that apply) 



□> 


District general funds 


128/ 




State funds for reading programs 


129/ 


□3 


State textbook funds 


130/ 


□4 


Title I 


131/ 


Ds 


Title II (Professional development to improve teacher quality) 


132/ 




Title III (Professional development for ELL teachers) 


133/ 


□ v 


Comprehensive School Reform 


134/ 


□ s 


2V^ Century Community Learning Centers 


135/ 




Reading Lirst 


136/ 


1 1 10 


Professional development funds 


137-138/ 


□ n 


Private grants 


139-140/ 



B2. Beyond financial support, has your school received external assistance this year (from district, 
state, publisher, university expert, etc.) implementing any of the following K-3 reading program 
activities? 





Check one box for each item... 




YES NO 


a. Selecting instmctional programs/materials 


□ 1 ^2 


b. Selecting assessment instmments 


□ 1 ^2 

142/ 


c. Selecting professional development providers 


□1 ^2 

143/ 


d. Conducting classroom observation 


□1 ^2 

144/ 


e. Conducting demonstration lessons 


□ 1 ^2 


f Interpreting assessment results 


□ 1 ^2 


g. Recmiting staff with reading expertise, e.g. teachers, coaches 


□ 1 Q 2 


h. Setting up intervention programs for struggling readers 


□ 1 Q 2 


i. Planning professional development 


□ 1 Q 2 


j. Providing technical assistance in implementing core reading program 


□ 1 Q 2 


k. Providing technical assistance for using supplementary reading materials 


□ 1 Q 2 


1 . Conducting needs assessment for professional development 


□ 1 Q 2 


m. Diagnosing needs of stmggling readers 


□ 1 Q 2 


n. Reviewing reading program effectiveness 


□ 1 Q 2 


0. Leading teacher study groups 


□ 1 ^2 
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C. Reading Instructional Materials 



Cl. Which core reading program is being used to teach reading in each of grades K-3 at this school? 
We have provided a partial list of eore reading programs below. Please eheck the core reading 
program used in each grade. If the core reading program you use is not included below, please 
check “other,” and write in the name of the program and the publisher and/or developer (if 
applieable). 

A core reading program is one that provides a comprehensive program of instruction on a daily basis in 
all aspects of reading. 

Note: Inclusion of a core reading program on the list below does not constitute an endorsement by 
the U.S. Department of Education. 



Publisher/ 

Developer 


Core reading program 


Check one per grade 
K 1 2 3 

156 - 157 / 158 - 159 / 160 - 161 / 162 - 163 / 




Unspecified or other (Please specify): 












Addison Wesley 






□l 


□1 


□1 


□1 






164 - 165 / 












Collections 




□2 


□2 


□2 


□2 




Rigby Reading 




□3 


□3 


□3 


□3 




Signatures 




□4 


□4 


□4 


□4 


Harcourt 


Trophies 




□5 


□5 


□5 


□5 




Unspecified or other (Please specify): 
















166 - 167 / 


□6 


□6 


□6 


□6 
















Fountas Pinnel units of study 




□7 


□7 


□7 


□7 


Heinemann 


Unspecified or other (Please specify): 
















168 - 169 / 


□s 


□8 


□8 


□s 
















Horizons 




□9 


□9 


□9 


□9 




Invitation to Literacy 




Qio 


Qio 


Qio 


Qio 




Lectura 




□11 


□11 


□11 


□11 




Legacy of Literacy 




□ 12 


□ 12 


□ 12 


□ 12 


Houghton Mifflin 


Nation's Choice 




□ l 3 


□ l 3 


□ l 3 


□ l 3 




Reading 




□ l 4 


□ l 4 


□ l 4 


□ l 4 




State Specific Edition 




□l 5 


□l 5 


□l 5 


□l 5 




Unspecified or other (Please specify): 
















170 - 171 / 


□16 


□16 


□16 


□16 
















Open Court 




□ 17 


□ 17 


□ 17 


□ 17 




Reading 




Qi 8 


Qi 8 


Qi 8 




McGraw-Hill 


Reading Mastery 




□ 19 


□ 19 


□ 19 


□19 




Spotlight on Literacy 




Q20 


Q20 


i::i 2 o 


1^20 




Unspecified or other (Please specify): 
















172 - 173 / 


□21 


□21 


□21 


□21 
















Saxon Phonics 




□22 


□22 


□22 


□22 


Saxon 


Unspecified or other (Please specify): 
















174 - 175 / 


Q23 


Q23 


Q23 


Q23 
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Cl. CONTINUED. Which core reading program is being used to teach reading in each of grades K-3 
at this school? We have provided a partial list of eore reading programs below. Please eheck the 
core reading program used in each grade. If the eore reading program you use is not included 
below, please check “other,” and write in the name of the program and the publisher and/or 
developer (if applieable). 

Note: Inclusion of a core reading program on the list below does not constitute an endorsement by 
the U.S. Department of Education. 



Publisher/ 

Developer 


Core reading program 


Check one per grade 
K 1 2 3 

156-157/ 158-159/ 160-161/ 162-163/ 




Literacy Place 


□24 I2I24 ^24 ^24 


Scholastic 


Unspecified or other (Please specify): 

176-177/ 


Q25 Q25 Q25 Q25 




Literacy Works 


Q26 ^26 ^26 ^26 




Reading 


□27 1^27 ^27 1^27 


Scott Foresman 


State Specific Edition 


Q28 1^28 1^28 1^28 




Unspecified or other (Please specify): 

178-179/ 


1^29 ^29 ^29 ^29 




Read Well 


□30 Oso Q30 Q30 


Sopris 


Unspecified or other (Please specify): 

180-181/ 


□31 Qai Qai Qai 




Success for All 


Q32 Q32 Q32 Q32 


Success for All 


Unspecified or other (Please specify): 

182-183/ 


1^33 1^33 ^33 ^33 




Universal Literacy 


i::i 34 i::i 34 Q34 1^:134 


Voyager 


Unspecified or other (Please specify): 

184-185/ 


□35 Gss Gss Qss 


Wright Group 


Unspecified or other (Please specify): 

186-187/ 


□36 Q36 Qse Q36 


Other 


Publisher/Developer: 188-189/ 

Program Title: i90-i9i/ 


1^37 1^37 1^37 ^37 


Publisher/Developer: 192-193/ 

Program Title: 194-195/ 


□ 38 ^38 ^38 ^38 


I use a core reading program developed by teachers or other school personnel 


□ 39 ^39 ^39 ^39 


I do not use a core reading program 


0 

□ 

0 

□ 

0 

n 

0 

□ 
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D. Instructional Time 



Dl. Please indieate for which grades your school has a scheduled reading block. 



A reading block is the time period that is formally scheduled for teaching reading. 



If yes , please indicate for how many minutes the reading block is scheduled. 
Does your school have a reading block in: 

Scheduled 
number of 

Yes minutes 

a. Kindergarten Q l ^ I I 

b. First grade Q i ► ! I 

c. Second grade Q i ► ! I 

d. Third grade Q i I 



D2. For which of the following activities are state staff, district staff, the principal, and the school 
reading coach responsible? 



No 

□ 

□ 

□ 

□ 



2 


196/ 


197-199/ 


2 


200/ 


201-203/ 


2 


204/ 


205-207/ 


2 


208/ 


209-211/ 





For each activity, check all that apply... 


School’s 

State District Principal reading 

coach 


N/A 


a. Selection of a specific core reading program 


1^1 212/ 1^2 213/ 1^3 214/ 1^4 215/ 


1^5 216/ 


b. Selection of supplemental reading program 
materials (for use with the whole class) 


1^1 217/ 1^2 218/ 1^3 219/ 1^4 220/ 


1^5 221/ 


c. Selection of intervention reading program 
materials (for use with struggling readers) 


1^1 222/ 1^2 223/ 1^3 224/ 1^4 225/ 


1^5 226/ 


d. Selection of reading assessment instruments 


1^1 227/ 1^2 228/ 1^3 229/ 1^4 230/ 


1^5 231/ 
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D3. This year, for which of the following activities are state staff, district staff, the principal, and the 
school reading eoach responsible? 



For each activity, check all that apply... 



Activities 














School’s 








State 




District 


Principal 


reading 

coach 


N/A 




a. Monitoring implementation of reading program 




232/ 




233/ 


□3 


234/ 


□4 


235/ 


□3 


236/ 


b. Review of teachers’ reading lesson plans 




237/ 




238/ 


□3 


239/ 


□4 


240/ 


□3 


241/ 


c. Review individual students’ progress in reading 




242/ 




243/ 


□3 


244/ 


□4 


245/ 


□3 


246/ 


d. Interpretation of assessment results 




247/ 




248/ 


□3 


249/ 


□4 


250/ 


□3 


251/ 


e. Feedback to teachers about reading instruction 




252/ 




253/ 


□3 


254/ 


□4 


255/ 


□3 


256/ 


f Selection of reading professional development 
topics and opportunities 




257/ 




258/ 


□3 


259/ 


□4 


260/ 


□3 


261/ 



D4. This sehool year, how often have you, as principal, evaluated K-3 reading instruetion using the 
following methods? 







Check only one box for each item ... 






Not at all 


1-4 times 
this school 
year 


5-8 times 
this school 
year 


Once a 
month 


Once a 
week or 
more 


a. Observed classroom reading instruction informally 




□3 


□3 


□4 


1^5 262/ 


b. Observed classroom reading instruction using an 
evaluation form 




□3 


□3 


□4 


1^5 263/ 


c. Met with teachers individually to discuss strategies 
for improving reading instruction 




□3 


□3 


□4 


1^5 264/ 


d. Met with groups of teachers to discuss strategies 
for improving reading instruction 




□3 


□3 


□4 


1^5 265/ 



D5. Has your school made any of the following changes to your reading program that took effeet at the 
beginning of the eurrent sehool year (2006-2007)? 





Check only one box for each item . 


•• 




YES 


NO 




a. Adopted a new core reading program 




□3 


266/ 


b. Added a new intervention program for struggiing readers 




□3 


267/ 


c. Added new supplementary materials 




□3 


268/ 


d. Added new materials for ELLs (see definition of ELL on page 

1) 




□3 


269/ 


I k Check if no ELLs 270/ 








e. Adopted new reading assessments 




□3 


271/ 
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E. Reading Interventions for Struggling Readers 



El . What methods has your school used to meet the needs of at-risk or struggling readers? For each 
method listed below, please check whether or not you use the method at your school. 



Reading intervention is a program designed for struggling readers to be used only with struggling 
readers in addition to the core-reading program. 







Check only one box for each 
item ... 


Methods of meeting needs of struggling readers 


Use this 
method 


Not used 




a. Use separate program materials in interventions 






272 / 




b. Use core reading program with supplemental materials 






273 / 




c. Use core reading program only 






274 / 


Materials 


d. Use reading materials written in ELLs’ home language (see definition of 
ELL on page 1) 






275 / 




e. Use alternative materials designed for ELLs 






276 / 




277 / 






f A certified reading specialist provides additional direct instruction to 
struggling readers, individually or in small groups. 






278 / 




g. The classroom teacher provides additional direct instruction to 
struggling readers, individually or in small groups. 






279 / 




h. The classroom teacher provides additional opportunities for reading skill 
practice for struggling readers (e.g., partner reading, peer tutors, audio 
tapes, computer programs). 






280 / 




i. A certified specialist provides recommendations to classroom teachers 
on accommodations for struggling readers. (Indicate which type of 
specialist.) 




staff 


A special education teacher 






281 / 


activities 


A bilingual/ESL teacher 






282 / 




Other (Please specify): 284 - 285 / 






283 / 




j. Trained aides or volunteers work with students under the direction of the 
classroom teacher during the school day. 






286 / 




k. Trained aides or volunteers work with students in a before or after 
school program. 






287 / 




1. Untrained aides or volunteers work with students under the direction of 
the classroom teacher during the school day. 






288 / 




m. Untrained aides or volunteers work with students in a before or after 
school program. 






289 / 
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E2. Which of the following methods has your sehool used this year to identify students for reading 
interventions in Grades K-3? 



Method of identifying students 


Check one box for each item... 


Use this method Not used 


a. Standardized achievement test scores in reading 


□. a 


b. Scores on tests that are part of the reading program 


□, a 

291 / 


c. Screening test scores in reading 


□, a 

292 / 


d. Diagnostic test scores in reading 


□, a 

293 / 


e. Progress monitoring test scores in reading 


□, a 

294 / 


f Documented classroom observations 


□, a 

295 / 


g. Teacher recommendations 


□, a 

296 / 


h. Other school staff recommendations 


□, a 

297 / 


i. Requests from parents 


□, a 

298 / 


j. Reading coach recommendation 


□, a 

299 / 



E3. 



Are reading intervention serviees (e.g., a reading speeialist) available this year to ehildren who 
need them? (Check ‘no’ if there are no ehildren who need intervention serviees) 

300 / 



□ i Yes 

□ 2 No 



E3a. If yes, What was the average wait for reading intervention services? 
sehool days. (Please enter ‘zero’ if there is no wait for serviees) 



301 - 303 / 
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F. Professional Development in Reading 



FI. Below is a list of topics that are often covered in professional development activities that are 
designed for building administrators. 

• In column A , identify the topics that were addressed in professional development activities in which you participated 

since July 1*^ of the current school year. 

• In column B , please identify the topics in which you need more professional development, whether or not your 
school’s professional development activities have covered these topics. 

• Please check all that apply in columns A and B. 







CHECK ALL THAT APPLY ... 


Topic 




A. 

Topics addressed iu 
professional 
development 


B. 

Topics in which I 
need more 
professional 
development 




a. Phonemic Awareness 




304/ 


□3 


305/ 


Content of 


b. Decoding 


□3 


306/ 


□4 


307/ 


Reading 


c. Vocabulary 




308/ 


□3 


309/ 


Instruction 


d. Fluency 


□3 


310/ 


□4 


311/ 




e. Comprehension 




312/ 


□3 


313/ 




f How to use the core reading program 




314/ 


□3 


315/ 




g. How to use children’s literature to teach reading 


□3 


316/ 


□4 


317/ 


Teaching 

Strategies 


h. How to use reading research to guide content of 
instruction 




318/ 


□3 


319/ 


i. How the core reading program incorporates research 
principles 


□3 


320/ 


□4 


321/ 




j. How to use the supplemental reading program 




322/ 


□3 


323/ 




k. How to integrate reading and writing instruction 




324/ 


□3 


325/ 




1. Strategies for teaching reading to ELLs 


□3 


326/ 


□4 


327/ 




m. How to evaluate a core reading program 




328/ 


□3 


329/ 


Evaluation 


n. How to evaluate reading instruction 


□3 


330/ 


□4 


331/ 


0 . How to coach teachers in reading instruction 




332/ 


□3 


333/ 




p. How to manage reading personnel 




334/ 


□3 


335/ 




q. How to diagnose reading problems 


□3 


336/ 


□4 


337/ 


Assessment 


r. How to administer assessments 




338/ 


□3 


339/ 




s. How to interpret and use assessment data to guide 
instruction 


□3 


340/ 


□4 


341/ 
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FI. CONTINUED. Below is a list of topics that are often covered in professional development 
activities that are designed for building administrators. 

• In column A , identify the topics that were addressed in professional development activities in which you participated 

since July 1*^ of the current school year. 

• In column B , please identify the topics in which you need more professional development, whether or not your 
school’s professional development activities have covered these topics. 

• Please check all that apply in columns A and B. 



(CONTINUED FROM PREVIOUS PAGE) 


CHECK ALL THAT APPLY ... 


Topic 




A. 

Topics addressed in 
professional 
development 


B. 

Topics in which I need 
more professional 
development. 




t. How to help struggling readers with decoding 


□3 


342/ 


□4 


343/ 




u. How to help struggling readers with vocabulary 




344/ 


□3 


345/ 


struggling 

Readers 


V. How to help struggling readers with comprehension 


□3 


346/ 


□4 


347/ 


w. How to motivate readers 




348/ 


□3 


349/ 




X. Strategies for teaching reading to students with diagnosed 
learning disabilities 


□3 


350/ 


□4 


351/ 




y. How to select reading materials 




352/ 


□3 


353/ 




z. How to use content standards for curriculum planning and 
teaching 


□3 


354/ 


□4 


355/ 


Organization, 
Management 
and Support 


aa. How to select reading assessments 




356/ 


□3 


357/ 


bb. Alignment of reading curriculum and instruction with 
state/district assessments 


□3 


358/ 


□4 


359/ 




cc. How to work with parents 




360/ 


□3 


361/ 




dd. Classroom management 


□3 


362/ 


□4 


363/ 



Thank you for your cooperation and for taking time to answer these questions. Please place the 
completed survey in the enclosed envelope, seal the envelope and return it to your evaluation liaison. 

If you have any questions about the survey, please call x-xxx-xxx-xxxx. 
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Abt ID / barcode here 


1-6/ 


0MB Number: 1875- 




BATCH: 7-9/ 


0232 



Reading First impiementation Study 
Reading Coach Survey 



The U.S. Department of Edueation’s Policy and Program Studies Service has contracted with Abt 
Associates Inc. to conduct a national evaluation of K-3 reading instruction in Reading First and Title I 
schools. The study’s data collection includes two rounds of survey administration (2005 and 2007) from 
both Reading First schools and Title I schools. Survey results from the 2005 administration involved 
over 9,000 school-based respondents, and they have been summarized in a recent report. The Reading 
First Evaluation: Interim Report (July 2006). 

In each survey administration, we ask the principal, reading coach, and a sample of K-3 teachers from 
each participating school to complete a questiormaire. Participants will help inform the U.S. Department 
of Education, Congress, policymakers, practitioners, and researchers about how K-3 reading instruction 
is implemented in schools and what strategies teachers use to provide high-quality, evidence-based 
reading instruction in grades K-3. 
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Additional Information 



The survey will take you approximately 30 minutes to complete. All responses to the survey will be 
kept confidential. All individual identifying information will be used only by persons on the research 
team. Information such as school location (state), participants’ general job titles, grades they teach, and 
gender will be included in the study data files to be submitted to the Department of Education. 
However, participants’ names will be stripped from all analysis data files and data files to be submitted 
to the Department of Education. We will not report any data about individual classrooms — all 
information will be reported at the grade and school levels. Neither your school nor your district will 
have access to any of the completed surveys at any time. 

Thank you for your cooperation with this survey! 



According to the Paperwork Reduetion Act of 1995, no persons are required to respond 
to a eolleetion of information unless such a collection displays a valid OMB control 
number. The valid OMB eontrol number for this information collection is 1875-0232. 
The time required to complete this information collection is estimated to average 30 
minutes per response, including the time to review instruction, search existing data 
resources, gather the data needed, and complete and review the information collection. 
If you have any comments concerning the accuracy of the time estimate or suggestions 
for improving this form, please write to: Policy and Program Studies Service, U.S. 
Department of Education, 400 Maryland Avenue, SW, Washington, DC, 20202. 



Instructions 



Unless otherwise noted, your responses should reflect your experiences during the 2006-2007 
school year in the school to which this survey was sent. 



■ Please complete all questions; each question includes directions for recording your answer. 



■ You are sometimes told to skip over some questions in the survey. When this happens, you will 
see an arrow with a note that tells you what question to answer like this: 

□ i Yes 

□2 No — Skip to E4 



■ If you have any questions about how to complete the survey, please call: x-xxx-xxx-xxxx. This 
is a free call and will connect you with our expert interviewers who can assist you. 
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A. 



Your Background and Experience 



A reading coach is a staff member whose primary role is to provide ongoing training and 
support to school staff in the delivery of effective reading instruction. 





Enter # below... 


A1 . Including this year, for how many years have you been the K-3 reading coach for this school? 
(If less than one year, enter 1.) 


years lo-ii/ 


A2. Including this year, for how many years have you worked at this school in any capacity? 
(If less than one year, enter 1.) 


years 12 - 13 / 


A3. Including this year, how many years of classroom experience do you have, as either a teacher 
and/or reading coach? (If less than one year, enter 1.) 




a. Number of years experience as a reading coach 


years 14 - 15 / 


b. Number of years experience as a teacher 


years 16 - 17 / 



B. Coach Responsibilities 





Enter number 
below... 


B 1 . This school year, for how many schools do you serve as the reading coach (including this 
school)? 


Schools 

18-19/ 


B2. This school year, for how many teachers do you serve as the reading coach (include all 
teachers in all schools)? 


Teachers 

20 - 21 / 


B3. Approximately what percentage of your time do you spend as the K-3 reading coach for 
this school? 


% 

22-24/ 
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B4. How central is each of the following activities to your work this year (since July at this 
school? 

Please rate the aetivity a “1” if you do not do the aetivity or if it is not at all eentral to your role as the literaey eoaeh. 
Rate the aetivity a “5” if it is absolutely eentral or eritieal to your work. 





Check only one box for each item. . . 


Activity 


Do not do 
or not at 
all central 




Somewhat 

central 




Absolutely 

central 


a. Administering/coordinating reading assessments 


□1 


□2 


□3 


□4 


□5 


25/ 


b. Compiling reading assessment data for teachers 


□1 


□2 


□3 


□4 


□5 


26/ 


c. Facilitating grade level meetings 


n. 




□3 






27/ 


d. Participating in school leadership team meetings 


n. 




□3 


^4 




28/ 


e. Facilitating or coordinating family literacy activities 


□1 


□2 


□3 


□4 


□5 


29/ 


f Ordering/managing reading instruction materials 


□1 


□2 


□3 


□4 


□5 


30/ 


g. Participating in professional development provided by 
the district, state or other consultants 


□1 


□2 


°3 


□4 


□5 


31/ 


h. Providing sub time for teachers to observe other more 
experienced teachers 


n. 


^2 


□ 3 


^4 




32/ 


i. Providing direct reading instruction to students 


n. 


^2 


□ 3 


^4 




33/ 


j. Providing training/professional development in reading 
materials, strategies, and assessments 


n. 


^2 


□ 3 


^4 




34/ 


k. Coaching staff on a range of topics (note: specific 
coaching activities are asked about in the next item) 


□1 


□2 


□3 


□4 


□5 


35/ 


1. Organizing professional development for K-3 teachers 


□1 


□2 


□3 


□4 


□5 


36/ 


m. Coordinating activities and meetings between 
classroom and special education teachers 


□1 


□2 


□3 


□4 


□5 


37/ 


n. Coordinating activities and meetings between 

classroom teachers and English Language Learner 
(ELL) staff 




^2 


□ 3 


^4 


^5 


38/ 


^ ► Check if no ELLs H ^ 

39/ 




0. Other (Please specify): 41-42/ 




^2 


□ 3 


^4 


^5 


40/ 



English language learner (ELL) indicates a student who is in the process of acquiring English and has a first 
language other than English. Other common related terms include language minority or limited English proficient 
(LEP) students, students in English as a second language (ESL), or students in classes for English for speakers of 
other languages (ESOL). 
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B5. When you coach K-3 staff, how central has each of the following activities been to your work 
this year (since July 1^^)? 

Please rate the activity a “1” if you do not do the activity or if it is not at all central to your role as the literacy 
coach. Rate the activity a “5” if it is absolutely central or critical to your work. 





Check only one box for each item. . . 


Coaching Activity 


Do not do 
or not at 
all central 




Somewhat 

central 




Absolutely 

central 


a. Giving demonstration lessons using core or supplemental 
materials 




^2 


□3 


^4 


^5 


43/ 


b. Assisting teachers in using the core program 






□3 


^4 




44/ 


c. Observing and providing feedback to teachers 




^2 


□3 


^4 




45/ 


d. Assisting teachers in forming instructional groups 




□2 


□3 


□4 


□5 


46/ 


e. Assisting teachers in designing strategies for addressing 
the needs of struggling readers 




^2 


□3 


^4 




47/ 


f Assisting teachers with monitoring the effectiveness of 
strategies addressing the needs of struggling readers 




^2 


□3 


^4 




48/ 


g. Giving demonstrations on assessment administration and 
scoring 




□2 


□3 


□4 


□5 


49/ 


h. Planning reading instruction with teachers 




□2 


□3 


□4 


□5 


50/ 


i. Reviewing teachers’ lesson plans and providing feedback 




^2 


□3 


^4 




51/ 


j. Assisting teachers in interpreting assessment results 




^2 


□3 


^4 




52/ 


k. Assisting teachers in designing strategies for addressing 
the needs of special education students 




^2 


□3 


^4 




53/ 


1. Assisting teachers in designing strategies for addressing 
the needs of ELLs (see page 2 for definition of ELL) 




□2 


□3 


□4 


□5 


54/ 


1 ^ Check if no ELLs □ i 

55/ 
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C. Reading Instructional Materials 



Cl. Which core reading program is being used to teach reading in each of grades K-3 at this school? 
We have provided a partial list of core reading programs below. Please check the core reading 
program used in each grade. If the core reading program you use is not included below, please 
check “other,” and write in the name of the program and the publisher and/or developer (if 
applicable). 

Note: Inclusion of a core reading program on the list below does not constitute an endorsement by 
the U.S. Department of Education. 



A core reading program is one that provides a comprehensive program of instruction on a daily basis in all 
aspects of reading. 



Publisher/ 

Developer 


Core reading program 


Check one per grade 

K 1 2 3 

56 - 57 / ^ 2 - 63 / 


Addison Wesley 


Unspecified or other (Please specify): 

64 - 65 / 


□i Qi Qi Qi 






Collections 


1^2 ^2 ^2 ^2 




Rigby Reading 


1^3 1^3 1^3 1^3 


Harcourt 


Signatures 


1^4 1^4 1^4 1^4 


Trophies 


□ 5 ^5 Qs ^5 




Unspecified or other (Please specify): 

66 - 67 / 


□6 Ue Q6 06 






Fountas Pinnel units of study 


1^7 1^7 1^7 1^7 


Heinemann 


Unspecified or other (Please specify): 

68 - 69 / 


Qg Qg Qg 






Horizons 


1^9 1^9 1^9 1^9 




Invitation to Literacy 


Qio QlO QlO QlO 




Lectura 


□ 11 Qn Qn Qn 




Legacy of Literacy 


□ 12 Qll Qi2 Qi2 


Houghton Mifflin 


Nation's Choice 


□ i3 Gis Qis Qis 




Reading 


□ l4 Qm Qm Qm 




State Specific Edition 


□ l5 ai5 ai5 ^15 




Unspecified or other (Please specify): 

70 - 71 / 


□ 16 ai6 Qie Qie 






Open Court 


□ 17 Qn Qn Qn 




Reading 


Qi8 ^18 ^18 ^18 


McGraw-Hill 


Reading Mastery 


□ 19 Qi 9 Qi 9 Qi 9 




Spotlight on Literacy 


1^20 Q 20 Q 20 Q 20 




Unspecified or other (Please specify): 

72 - 73 / 


□21 ^21 Q 21 i::i 2 i 






Saxon Phonics 


Q 22 ^22 ^22 ^22 


Saxon 


Unspecified or other (Please specify): 

74 - 75 / 


1^23 ^23 ^23 ^23 
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Cl. CONTINUED. Which core reading program is being used to teach reading in each of grades K-3 
at this school? We have provided a partial list of eore reading programs below. Please eheck the 
core reading program used in each grade. If the eore reading program you use is not included 
below, please check “other,” and write in the name of the program and the publisher and/or 
developer (if applieable). 

Note: Inclusion of a core reading program on the list below does not constitute an endorsement by 
the U.S. Department of Education. 



Publisher/ 

Developer 


Core reading program 


Check one per grade 
K 1 2 3 




Literacy Place 


Q24 Q24 Q24 Q24 


Scholastic 


Unspecified or other (Please specify): 

76-77/ 


Q25 Q25 Q25 Q25 




Literacy Works 


^26 ^26 ^26 ^26 




Reading 


Q27 1^27 1^27 1^27 


Scott Foresman 


State Specific Edition 
Unspecified or other (Please specify): 

78-79/ 


Q28 1^28 1^28 1^28 

Q29 ^29 ^29 ^29 




Read Well 


□30 Qso Q30 Q30 


Sopris 


Unspecified or other (Please specify): 

80-81/ 


□31 Gsi Gsi Gsi 


Success for All 


Success for All 

Unspecified or other (Please specify): 

82-83/ 


Q32 Q32 Q32 Q32 

1^33 1^33 ^33 ^33 




Universal Literacy 


i::i 34 i::i 34 Q34 1^:134 


Voyager 


Unspecified or other (Please specify): 

84-85/ 


□ 35 ^35 ^35 ^35 


Wright Group 


Unspecified or other (Please specify): 

86-87/ 


□ 36 ^36 ^36 Q36 


Other 


Publisher/Developer: 88-89/ 

Program Title: 90-9i/ 


□ 37 ^37 ^37 ^37 


Publisher/Developer: 92-93/ 

Program Title: 94-95/ 


Q38 i::i 38 i::i 38 1-I38 


I use a core reading program developed by teachers or other school personnel 


□ 39 ^39 ^39 ^39 


I do not use a core reading program 


□40 ^40 ^40 ^40 
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C2. Has your school made any of the following changes to your reading program that took effect at 
the beginning of the current school year (2006-2007)? 





Indicate ^yes’ or 


^no’ for each item . 


.. 




Yes 


No 




a. Adopted a new core reading program 






96/ 


b. Added a new intervention program for struggling readers 






97/ 


c. Added new supplementary materials 






98/ 


d. Added new materials for ELLs (see definition of ELL on page 2) 






99/ 


e. Adopted new reading assessments 






101/ 



D. Instructional Time 



Dl. Please indicate for which grades your school has a scheduled reading block. 



A reading block is the time period that is formally scheduled for teaching reading. 



If yes, please indicate for how many minutes the reading block is scheduled. 
Does your school have a reading block in: 



Scheduled 
number of 





Yes 


minutes 


No 




a. Kindergarten 


□ i 


1 


□ 2 


102/ 103-105/ 


b. First grade 


□ i 


-1 1 


□ 2 


106/ 107-109/ 


c. Seeond grade 


□ i 


-1 1 


□ 2 


110/ 111-113/ 


d. Third grade 


□ i 


— 


□ 2 


114/ 115-117/ 
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E. Reading Interventions for Struggling Readers 



El . What methods has your school used to meet the needs of at-risk or struggling readers? For 
each method listed below, please check whether or not you use the method at your school. 



A reading intervention is a program designed for struggling readers to be used only with struggling readers in 
addition to the core-reading program. 







Check one for each item 




Methods for meeting needs of struggling readers 


Use this 
method 


Not used 




a. Use separate program materials in interventions 




□2 


118/ 




b. Use core reading program with supplemental materials 




□2 


119/ 




c. Use core reading program only 




□2 


120/ 


Materials 


d. Use reading materials written in ELLs’ home language (see definition of 
ELL on page 2) 






121/ 




e. Use alternative materials designed for ELLs 


□1 


^2 


122/ 




1 ^heck if no ELLs H ^ 123 / 






f A certified reading specialist provides additional direct instruction to 
struggling readers, individually or in small groups. 


□1 


□2 


124/ 




g. The classroom teacher provides additional direct instruction to struggling 
readers, individually or in small groups. 


□1 


□2 


125/ 




h. The classroom teacher provides additional opportunities for reading skill 
practice for struggling readers (e.g., partner reading, peer tutors, audio 
tapes, computer programs). 




^2 


126/ 




i. A certified specialist provides recommendations to classroom teachers on 
accommodations for struggling readers. (Indicate which tvpe of 
specialist.) 




staff 

activities 


A special education teacher 


□1 


^2 


127/ 


A bilingual/ESL teacher 


□1 


^2 


128/ 




Other (Please specify): 130 - 131 / 




□2 


129/ 




j. Trained aides or volunteers work with students under the direction of the 
classroom teacher during the school day. 


□1 


□2 


132/ 




k. Trained aides or volunteers work with students in a before or after school 
program. 




^2 


133/ 




1. Untrained aides or volunteers work with students under the direction of the 
classroom teacher during the school day. 




^2 


134/ 




m. Untrained aides or volunteers work with students in a before or after 
school program. 


□1 


□2 


135/ 
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F. Professional Development for Reading Coaches 



FI. Below is a list of professional development topies for reading coaches in which you may have 
participated. 

• In column A , identify any topics that were addressed in reading coaches’ professional development activities 
during the current school year, including summer 2006 . 

• In column B , please identify the topics in which you need more professional development, whether or not this 
school’s professional development activities have covered these topics. 

• Please check all that apply in columns A and B. 

I CHECK ALL THAT APPLY 



Topics 


A. Topics addressed in 
professional development 
for reading coaches 


B. Topics in which you 
need more professional 
development 


a. How to use reading assessment data to guide instruction. 




136/ 


□3 


137/ 


b. What are the types of assessments: screening, diagnostic, progress 
monitoring, and outcome? 


□3 


138/ 


□4 


139/ 


c. How to use assessment data to form instructional groups. 




140/ 


□3 


141/ 


d. How to provide constructive feedback to teachers. 


□3 


142/ 


□4 


143/ 


e. How to establish credibility with teachers. 




144/ 


□3 


145/ 


f Essential components of scientifically based reading instruction. 


□3 


146/ 


□4 


147/ 


g. What is the role of the reading coach in fostering change? 




148/ 


□3 


149/ 


h. How to plan instructional interventions for struggling students. 


□3 


150/ 


□4 


151/ 


i. Classroom management within the literacy block time. 




152/ 


□3 


153/ 


j. How to conduct effective grade level meetings. 


□3 


154/ 


□4 


155/ 


k. How to help teachers identify appropriate instructional materials. 




156/ 


□3 


157/ 


1. How to help teachers make reading instruction systematic and 
explicit. 


□3 


158/ 


□4 


159/ 


m. How to conduct demonstration lessons. 




160/ 


□3 


161/ 


n. How to conduct classroom observations. 


□3 


162/ 


□4 


163/ 


0 . How to provide onsite professional development. 




164/ 


□3 


165/ 


p. How to provide instructional supports for ELL students learning to 
read (see definition of ELL on page 2). 


□3 


166/ 


□4 


167/ 


^ Check if no ELLs 1^ i 168 / 
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G. Reading Instruction 



Gl. This item asks you to describe your school using the statements below. Please read each 

statement, and indicate whether the statement is a good description of your school on a scale 
from a “Very inaccurate” description of your school to a “Very accurate” description of your 
school. 



Check one box for each item... 



In this school ... 


Very 

inaccurate 


















— ^ vciy 

accurate 


a. K-3 teachers are knowledgeable about scientifically based reading 
instruction. 




□2 


□3 


□4 


1^5 169/ 


b. K-3 teachers are motivated to improve reading instruction. 






□ 3 




□5 170/ 


c. Reading instruction in K-3 classrooms is aligned with the state 
reading/language arts content standards. 






□ 3 


^4 


□5 171/ 


d. There is a school-wide focus on reading and language arts. 




□2 


□3 


□4 


172/ 


e. K-3 teachers are experienced with the core reading program. 




□2 


□3 


□4 


□5 173/ 


f K-3 teachers are experienced with supplemental reading materials. 




^2 


□ 3 


^4 


□5 174/ 


g. K-3 teachers are experienced with reading intervention materials and 
strategies. 




□2 


□3 


□4 


□5 175/ 


h. K-3 classrooms have ample, high quality instructional materials. 




□2 


□3 


□4 


176/ 


i. Teachers use a variety of instructional materials to fill in gaps in the 
core program. 




^2 


□ 3 


^4 


□5 177/ 


j. The core reading program is aligned with scientifically based reading 
research. 




^2 


□ 3 


^4 


□5 178/ 


k. Supplemental reading materials are aligned with scientifically based 
reading research. 




^2 


□ 3 


^4 


□5 179/ 


1. Reading intervention materials are aligned with scientifically based 
reading research. 




□2 


□3 


□4 


180/ 


m. The reading coach has the support of the school principal. 




□2 


□3 


□4 


□5 181/ 


n. K-3 teachers seek the assistance of the reading coach to improve their 
reading instruction. 




^2 


□ 3 


^4 


□5 182/ 


0 . Sufficient time during the school day is allotted for reading instruction. 




^2 


□ 3 


^4 


1^5 183/ 


p. Sufficient time during the school day is allotted for teacher planning. 




□2 


□3 


□4 


184/ 


q. K-3 teachers collaborate and plan for reading instruction. 




□2 


□3 


□4 


□5 185/ 


r. Sufficient time during the school day is allotted for professional 
development. 




^2 


□ 3 


^4 


□5 186/ 


s. Reading assessments are used to screen students for reading difficulties. 




°2 


□ 3 


^4 


□5 187/ 


t. Diagnostic assessments are used to identify strengths and weaknesses of 
struggling readers. 




□2 


□3 


□4 


188/ 


u. Reading assessments are used to monitor student progress. 




□2 


□3 


□4 


189/ 


V. Assessment data are used to group students for instruction. 




^2 


□ 3 


^4 


□5 190/ 


w. Assessment data are used to guide and/or modify instruction. 




□2 


□3 


□4 


191/ 


X. The district provides direction concerning reading instruction. 




□2 


□3 


□4 


192/ 


y. The state provides direction concerning reading instruction. 




°2 


□ 3 


^4 


□5 193/ 


z. K-3 teachers make an effort to involve parents in their children’s 
reading instruction. 




□2 


□3 


□4 


194/ 
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Thank you for your cooperation and for taking time to answer these questions. Please place the 
completed survey in the enclosed envelope, seal the envelope and return it to your evaluation liaison. 

If you have any questions about the survey, please call x-xxx-xxx-xxxx. 
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Construct Variables for Analysis of Survey Data 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


Item Scale 


Scoring Metric 


Support for Struggling Readers 


Support for 

Struggling 

Readers^ 


Grade 1 
Teacher 
Survey, C9 


b 


Grade 1 
Teacher 
Survey, 
B8 


b 


Extra practice with phonemic awareness 


Dichotomous 
1 = “Received” 
0 = “Did not 
receive” 


Score 1 for each 
Survey Item checked 
as “Received.” 

Sum Survey Item 
scores 

(Range: 0-3) 


c 


c 


Extra practice with decoding/phonics 


d 


d 


Extra practice with fluency 


High Fidelity (SBRR) Reading Instruction— KINDERGARTEN (K) 


KINDERGARTEN 
High Fidelity 
Vocabulary 
Instruction 


K Teacher 
Survey, C4 


bb. 


K 

Teacher 

Survey, 

B4 


bb. 


I discuss meaning of new and unusual 
words 


Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction” 


Score 1 if Survey 
Item checked 
“Central to my 
instruction.” 

Compute percent of 
items in construct 
rated as “central.” 

(Range: 0 to 100%) 


dd. 


dd. 


Students give definitions for words 


ee. 


ee. 


Students tell opposites of words 



(Continues) 



86 



We created two versions of several constructs: a ‘strict’ version and a more ‘relaxed’ version. Our hope is to use the ‘strict’ version of the construct, as we 
believe these represent the highest quality activities. However, it could be the case that we find little variation across these strict constructs, especially in our 
sample of new RF schools. Therefore, the more relaxed versions may be more appropriate for analysis as they allow for more variation across respondents. 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 2005 Survey 2007 Survey Survey Description 

High Fideiity (SBRR) Reading Instruction — KiNDERGARTEN (K) (continued) 



KINDERGARTEN 
High Fidelity 
Comprehension 
Instruction 



K Teacher 
Survey, C4 



b. K Teacher 
Survey, B4 



KINDERGARTEN K Teacher 
Non-SBRR Instruction Survey, C4 



K Teacher 
Survey, B4 



I conduct story discussions with small groups of 
students 

I read stories to small groups of students 



Students orally answer questions about stories 
they have heard 

Students read texts that are easy to decode 

Students create story maps based on stories 

read aloud 

Students retell stories in sequence and identify 

characters and main events 

Students make predictions while reading stories 

I read stories aloud to the whole class 

I develop language experience stories with my 

class 

Students read aloud their own written dictation 
I teach phonemic awareness skills while reading 

stories 

Students practice naming letters 

I teach sight words 

Students tell opposites of words 

Students use dictionaries to find word meanings 



Item Scale 



Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction” 



Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction” 



Scoring Metric 



Score 1 for each 
Survey Item 
checked “Central 
to my instruction.” 

Compute percent 
of items in 
construct rated as 
“central.” 

(Range; 0 to 
100%) 



Score 1 for each 
Survey Item 
checked “Central 
to my instruction.” 

Sum Survey Item 



(Range: 0 to 9) 



(Continues) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


Item Scale 


Scoring Metric 


High Fideiity (SBRR) Reading Instruction — KiNDERGARTEN (K) (continued) 


KINDERGARTEN 
High Fidelity 
Phonemic 

Awareness/Phonics 

Instruction 


K Teacher 
Survey, C4 


m. 


K Teacher 
Survey, B4 


m. 


Students isolate first sounds in words that I say 


Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction” 


Score 1 for each 
Survey Item 
checked “Central 
to my instruction.” 

Sum Survey Item 

scores 

Compute percent 
of Items in 
construct rated as 
“central.” 

(Range: 0 to 
100%) 


n. 


n. 


Students isolate final sounds in words that I say 


0. 


0. 


Students isolate middle sounds in words that I 
say 


p- 


p- 


Students blend sounds with rhyming words 


r. 


r. 


Students match sounds with letters 


s. 


s. 


Students blend sounds to form words 


t. 


t. 


I practice identifying sounds and syllables in 
spoken words by clapping and counting 


KINDERGARTEN 
High Fidelity Teaching 
Strategies 


K Teacher 
Survey, C6 


a. 


K Teacher 
Survey, B4 


a. 


Provide time in reading block for skill practice on 
own 


Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction” 


Score 1 for each 
Survey Item 
checked “Central 
to my instruction.” 

Compute percent 
of items in 
construct rated as 
“central.” 

(Range: 0 to 
100%) 


c. 


c. 


Provide extra reading instructional time for 
struggling readers 


i. 


i. 


Work with small groups of students 


j- 


j- 


Group students based on skill levels 


m. 


m. 


Use core reading series 


n. 


n. 


Use supplementary reading materials 


r. 


r. 


Use separate intervention materials for some 
students 


u. 


u. 


Use test results to organize instructional groups 


w. 


w. 


Use tests to determine progress on skills 


y- 


y- 


Use diagnostic tests to identify students who 
need reading intervention services 



(Continues) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


Item Scale 


Scoring Metric 


High Fideiity (SBRR) Reading Instruction — GRADE 1 


GRADE 1 High 
Fidelity Fluency 
Instruction 


Grade 1 
Teacher 
Survey, C4 


d. 


Grade 1 
Teacher 
Survey, B4 


d. 


Students reread familiar stories 


Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction” 


Score 1 for each 
Survey Item 
checked “Central to 
my instruction.” 

Sum Survey Item 
scores 

Compute percent of 
Items In construct 
rated as “central.” 

(Range; 0 to 100%) 


h. 


h. 


Students read aloud with expression and proper 
phrasing 


GRADE 1 High 
Fidelity Vocabulary 
Instruction (No 
construct — only 
individual item) 


Grade 1 
Teacher 
Survey, C4 


Z. 


Grade 1 
Teacher 
Survey, B4 


z. 


I discuss new and unusual words before reading 


Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction 


Score 100% if 
Survey Item 
checked “Central to 
my instruction.” 

Score 0% if item 
scored 2 or 3. 
(Range: 0 to 100%) 


GRADE 1 Non-SBRR 
Instruction 


Grade 1 
Teacher 
Survey, C4 


c. 


Grade 1 
Teacher 
Survey, B4 


c. 


Students read silently 


Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction” 


Score 1 for each 
Survey Item 
checked “Central 
to my instruction.” 

Sum Survey Item 
scores 

Compute percent 
of Items in 
construct rated as 
“central.” 

(Range; 0 to 100%) 


m. 


m. 


Students practice naming letters 


p- 


p- 


Students use knowledge of root words, prefixes, 
and suffices to decode new words 


r. 


r. 


Students use pictures to identify unknown words 


s. 


s. 


I teach decoding skills while reading stories 



m 

■ 

Ol 
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Construct Variabies for Anaiysis of Survey Data (Continued) 
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Construct Variables for Analysis of Survey Data (Continued) 



Construct 2005 Survey 2007 Survey Survey Description 

High Fidelity (SBRR) Reading Instruction — GRADE 1 (continued) 

GRADE 1 High Fidelity Grade 1 i. Grade 1 Teacher i. Students reread to find facts to answer questions 

Comprehension Teacher Survey, B4 

Instruction Survey, C4 

w. w. Students retell stories in sequence and identify 

characters and main events 



Survey Description 



cc. Students make predictions while reading stories 



Item Scale 

Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 
= “Not part of my 
reading instruction” 



Scoring Metric 



Score 1 for each 
Survey Item checked 
“Central to my 
instruction.” 

Sum Survey Item 



Compute percent of 
items in construct 
rated as “central.” 



GRADE 1 High Fidelity 
Phonemic 

Awareness/Phonics 

Instruction 



Grade 1 
Teacher 
Survey, C4 



ff. Students develop questions about text material 



I. Grade 1 Teacher I. Students isolate sounds in words that teachers say 
Survey, B4 

n. n. Students blend phonemes to form words 



u. Teaches decoding/phonics skills with word families 



Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 
= “Not part of my 
reading instruction” 



(Range: 0 to 100%) 



Score 1 for each 
Survey Item checked 
“Central to my 
instruction.” 

Sum Survey Item 



Compute percent of 
items in construct 
rated as “central.” 



GRADE 1 High Fidelity 
Teaching Strategies 



Grade 1 
Teacher 
Survey, C6 



Grade 1 Teacher 
Survey, B4 



Provide time in reading block for skill practice on own 

Provide extra reading instructional time for struggling 

readers 

Work with small groups of students 

Group students based on skill levels 

Use core reading series 

Use supplementary reading materials 

Use separate intervention materials for some students 

Use test results to organize instructional groups 

Use tests to determine progress on skills 

Use diagnostic tests to identify students who need 
reading intervention services 



Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 
= “Not part of my 
reading instruction” 



(Range: 0 to 100%) 
Score 1 for each 
Survey Item checked 
“Central to my 
instruction.” 

Sum Survey Item 



Compute percent of 
items in construct 
rated as “central.” 

(Range: 0 to 100%) 

(Continues) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


Item Scale 


Scoring Metric 


High Fideiity (SBRR) Reading Instruction — GRADE 2/3 


GRADE 2/3 High 
Fidelity Fluency 
Instruction 


Grade 2/3 
Teacher 
Survey, C4 


d. 


Grade 2/3 
Teacher 
Survey, B4 


d. 


Students reread familiar stories 


Three-level ordinal 

1 = “Central to my 
reading instruction” 


Score 1 for each 
Survey Item 
checked “Central 
to my instruction.” 






g- 




g- 


I listen to students read aloud without correcting 
errors 


2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction” 


Sum Survey Item 
scores 

Compute percent 
of items in 
construct rated as 
“central.” 
















(Range: 0 to 
100%) 


GRADE 2/3 High 
Fidelity Vocabulary 
Instruction 


Grade 2/3 
Teacher 
Survey, C4 


t. 


Grade 2/3 
Teacher 
Survey, B4 


t. 


Students work with prefixes and suffixes to 
change the meaning of words 


Three-level ordinal 

1 = “Central to my 
reading instruction” 

2 = “Small part of 
reading instruction” 

3 = “Not part of my 
reading instruction” 


Score 1 if Survey 
Item checked 
“Central to my 
instruction.” 

Sum Survey Item 
scores 






X. 




X. 


Students learn vocabulary through study of 
antonyms, synonyms, and homonyms 






bb 




bb. 


I discuss new and unusual words before reading 


Compute percent 
of items in 
construct rated as 
“central.” 
















(Range: 0 to 
100%) 



(Continues) 
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Construct 2005 Survey 2007 Survey Survey Description 

High Fideiity (SBRR) Reading Instruction — GRADE 2/3 (continued) 

GRADE 2/3 Non- Grade 2/3 a. Grade 2/3 a. Students read texts that are easy to decode 
SBRR Instruction Teacher Teacher 



Construct Variabies for Anaiysis of Survey Data (Continued) 



GRADE 2/3 High 
Fidelity 

Comprehension 

Instruction 



Survey, C4 



Survey, B4 



Grade 2/3 
Teacher 
Survey, C4 



b. Grade 2/3 
Teacher 
— Survey, B4 



c. Students read aloud unfamiliar texts 

f. Students read silently 

0. I teach decoding skills while reading stories 

p. Students memorize sight words 

V. Students use context clues to identify unknown 

words 

z. Students write vocabulary words in sentences 

aa. Students use dictionaries to find word meanings 
b. Students reread to locate information 

1. Students confirm or revise predictions after reading 



Students generate their own questions about text 
material 

Students identify their comprehension break-downs 
and use fix-up strategies with a partner 

Students orally summarize main events in stories 
and informational texts 

I discuss new and unusual words before reading 



Item Scale 

Three-level 

ordinal 

1 = “Central to 
my reading 
instruction” 

2 = “Small part 
of reading 
instruction” 3 = 
“Not part of my 
reading 
instruction” 



Three-level 

ordinal 

1 = “Central to 
my reading 
instruction” 

2 = “Small part 
of reading 
instruction” 3 = 
“Not part of my 
reading 
instruction” 



Scoring Metric 

Score 1 for each 
Survey Item 
checked “Central to 
my instruction.” 

Sum Survey Item 



(Compute percent 
of items in 
construct rated as 
“central.” 

(Range: 0 to 100%) 



Score 1 for each 
Survey Item 
checked “Central to 
my instruction.” 

Sum Survey Item 



Compute percent of 
items in construct 
rated as “central.” 

(Range: 0 to 100%) 



dd. Students identify story structure and elements 



(Continues) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



2005 Survey 


2007 Survey 


Survey Description 


Item Scaie 



Construct 



Scoring Metric 



High Fideiity (SBRR) Reading Instruction — GRADE 2/3 (continued) 



GRADE 2/3 High 
Fidelity Phonemic 
Awareness/Phonics 
Instruction 


Grade 

2/3 

Teacher 

Survey, 

C4 


n. 


Grade 2/3 
Teacher 
Survey, B4 


n. 


Students decode multi-syllabic words in isolation 


Three-level 

ordinal 

1 = “Central to 
my reading 
instruction” 

2 = “Small part 
of reading 
instruction” 3 = 
“Not part of my 
reading 
instruction” 


Score 1 for each 
Survey Item 
checked “Central 
to my instruction.” 

Score -1 for each 
distractor checked 
“Central to my 
instruction.” 

Sum Survey Item 
scores 

Compute percent 
of items in 
construct rated as 
“central.” 

(Range: 0 to 
100%) 


q- 


q- 


Students read irregularly spelled words and non- 
words 


S. 


S. 


Students use knowledge of root woods, prefixes, 
and suffixes to decode new words 


GRADE 2/3 High 


Grade 


a. 


Grade 2/3 


a. 


Provide time in reading block for skill practice on 


Three-level 


Score 1 for each 


Fidelity Teaching 


2/3 




Teacher 




own 


ordinal 


Survey Item 


Strategies 


Teacher 


c. 


Survey, B4 


c. 


Provide extra reading instructional time for 




checked “Central 




Survey, 








struggling readers 


1 = “Central to 


to my instruction.” 




C6 


i. 




i. 


Work with small groups of students 


my reading 








j- 




j- 


Group students based on skill levels 


instruction” 


Sum Survey Item 






m. 




m. 


Use core reading series 


2 = “Small part 


scores 






n. 




n. 


Use supplementary reading materials 


of reading 








r. 




r. 


Use separate intervention materials for some 


instruction” 3 = 


Compute percent 












students 


“Not part of my 


of items in 






u. 




u. 


Use test results to organize instructional groups 


reading 


construct rated as 






w. 




w. 


Use tests to determine progress on skills 


instruction” 


“central.” 






y- 




y- 


Use diagnostic tests to identify students who need 
















reading intervention services 




(Range: 0 to 
100%) 



(Continues) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


item Scaie 


Scoring Metric 


Professional Development (PD) for Teachers (Applies to all q 


irade ieveis) 


Received/Need More 
PD in Phonemic 
Awareness 


Teacher 
Survey, D, 
D4 


a. 


Teacher 
Survey, C4 


a 


Building phonological awareness 


Dichotomous 
1 = “Topic addressed in 
Professional 
Development” 

0 = “Topic not 
addressed” OR 

1 = “Topic in which I 
need more Professional 
Development” 

0 = “Topic in which I do 
not need more PD” 


Score 1 if any of 
Survey Items a - c 
are checked “Topic 
addressed in PD.” 
OR “Topic in which I 
need more PD” 

(Range: 0 to 1) 


b. 


b 


Identifying, adding, deleting sounds in spoken 
words 


c. 


c 


Blending phonemes to form words 


Received/Need More 
PD in Decoding 


Teacher 
Survey, D, 
D4 


d. 


Teacher 

Survey, 

C4 


d 


Teaching letter-sound correspondence 


Dichotomous 
1 = “Topic addressed in 
Professional 
Development” 

0 = “Topic not 
addressed” OR 
1= “Topic in which I need 
more Professional 
Development” 

0= “Topic in which I do 
not need more PD” 


Score 1 if any of 
Survey Items d - g 
are checked “Topic 
addressed in PD.” 
OR “Topic in which I 
need more PD” 

(Range: 0 to 1) 


e. 


e 


Teaching letter patterns 


f. 


f 


Using syllable patterns to read words 


g- 


g 


Teaching component parts 


Received/Need More 
PD in Vocabulary 


Teacher 
Survey, D4 


h. 


Teacher 
Survey, C4 


h 


Teaching use of dictionary, thesaurus 


Dichotomous 
1 = “Topic addressed 
in Professionai 
Deveiopment” 

0 = “Topic not 
addressed” OR 

1 = “Topic in which i 
need more 
Professionai 
Deveiopment” 

0 = “Topic in which i 
do not need more PD” 


Score 1 if any of 
Survey Items h - j 
are checked 
“Topic addressed 
in PD.” OR “Topic 
in which i need 
more PD” 

(Range: 0 to 1) 


i. 


i 


Direct teaching of vocabulary words and 
meaning 


j- 


j 


Antonyms and synonyms 



(Continues) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


Item Scale 


Scoring Metric 


Professional Development (PD) for Teachers (continued) 


Received/Need More 
PD in Fluency 


Teacher 
Survey, D4 


k. 


Teacher 
Survey, C4 


k 


Teaching sight words 


Dichotomous 
1 = “Topic addressed 
in Professionai 
Deveiopment” 

0 = “Topic not 
addressed” OR 

1 = “Topic in which i 
need more 
Professionai 
Deveiopment” 

0 = “Topic in which i 
do not need more 
PD” 


Score 1 if any of 
Survey Items k 
m are checked 
“Topic addressed 
in PD.” OR “Topic 
in which i need 
more PD” 

(Range: 0 to 1) 


I. 


I 


Guided orai reading 


m. 


m 


Encouraging expression while reading 


Received/Need More 
PD in 

Comprehension 


Teacher 
Survey, D4 


n. 


Teacher 
Survey, C4 


n 


Setting motivation/asking prediction/preview 
questions 


Dichotomous 
1 = “Topic addressed 
in Professionai 
Deveiopment” 

0 = “Topic not 
addressed” OR 

1 = “Topic in which i 
need more 
Professionai 
Deveiopment” 

0 = “Topic in which i 
do not need more 
PD” 


Score 1 if any of 
Survey Items n 
s are checked 
“Topic addressed 
in PD.” OR “Topic 
in which i need 
more PD” 

(Range: 0 to 1) 


0. 


0 


Constructing information about character, 
setting, and main events 


p- 


p 


Summarizing main ideas in narrative and 
informationai text 


q- 


q 


Self-monitoring strategies 


r. 


r 


Asking questions at different leveis (literai, 
inferentiai) 


s. 


s 


Strategies for organizing text structure, e.g. 
story maps 



(Continues) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


Item Scale 


Scoring Metric 


Professional Development (PD) for Teachers (continued) 


Received/Would Like 
More PD in Teaching 
Strategies 


Teacher 
Survey, D5 


a. 


Teacher 
Survey, C5 


a 


How to use the core reading program 


Dichotomous 
1 = “Topic addressed in 
Professional 
Development” 

0 = “Topic not 
addressed” OR 

1 = “Topics in which I’d 
like more Professional 
Development” 

0 = “Topic in which I 
would not like more PD” 


Score 1 if any of 
Survey Items a - g 
are checked “Topic 
addressed in PD.” 
OR “Topic in which 
I’d like more PD” 
(Range: 0 to 1 ) 


b. 


b 


How to use children’s literature to teach 
reading 


c. 


c 


How to use reading research to guide 
content of instruction 


d. 


d 


How the core reading program incorporates 
research principles 


e. 


e 


How to use the supplemental reading 
program(s) 


f. 


f 


How to integrate reading and writing 
instruction 


g- 


g 


Strategies for teaching reading to ELLs 


Received/Would Like 
More PD in Grouping 


Teacher 
Survey, D5 


h. 


Teacher 
Survey, C5 


h 


Learning styles 


Dichotomous 
1 = “Topic addressed in 
Professional 
Development” 

0 = “Topic not 
addressed” OR 

1 = “Topics in which I’d 
like more Professional 
Development” 

0 = “Topic in which I 
would not like more PD” 


Score 1 if any of 
Survey Items h - i 
are checked “Topic 
addressed in PD.” 
OR “Topic in which 
I’d like more PD” 

(Range: 0 to 1 ) 


i. 


i 


How to organize small group instruction 


Received/Would Like 
More PD in 
Assessment 


Teacher 
Survey, D5 


j- 


Teacher 
Survey, C5 


j- 


How to diagnose reading problems 


Dichotomous 
1 = “Topic addressed in 
Professional 
Development” 

0 = “Topic not 
addressed” OR 

1 = “Topics in which I’d 
like more Professional 
Development” 

0 = “Topic in which I 
would not like more PD” 


Score 1 if any of 
Survey Items j - 1 
are checked “Topic 
addressed in PD.” 
OR “Topic in which 
I’d like more PD” 

(Range: 0 to 1 ) 


k. 


k 


How to administer assessments 


I. 


I. 


How to interpret and use assessment data 
to guide instruction 



(Continues) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 
Construct 2005 Survey 2007 Survey 



I I 

Professional Development (PD) for Teachers (continued) 


Received/Would Like 
More PD in 


Teacher 
Survey, D5 


m. 


Teacher 
Survey, 05 


m. 


Struggling Readers 




n. 




n. 






0. 




0. 






p- 




p- 






q- 




q- 


Received/Would Like 
More PD in 


Teacher 
Survey, D5 


r. 


Teacher 
Survey, 05 


r. 


Organization/ 

Planning 




s. 




s. 






t. 




t. 






u. 




u. 


Number of PD items 
received in Teaching 


Teacher 
Survey, D5 


a. 


Teacher 
Survey, 05 


a 


Strategies 




b. 




b 






c. 




c 






d. 




d 






e. 




e 






f. 




f 






g- 




g 



Survey Description 



Item Scaie 



Scoring Metric 



How to help struggling readers with decoding 

How to help struggling readers with 
vocabulary 

How to help struggling readers with 
comprehension 

How to motivate readers 

Strategies for teaching reading to students 
with diagnosed learning disabilities 

How to use state/district content standards for 
curriculum planning and teaching. 

How to align reading curriculum and 
instruction with state/district assessments 

How to work with parents 
Classroom management 



How to use the core reading program 

How to use children’s literature to teach 
reading 

How to use reading research to guide content 
of instruction 

How the core reading program incorporates 
research principles 
How to use the supplemental reading 
program (s) 

How to integrate reading and writing 

instruction 

Strategies for teaching reading to ELLs 



Dichotomous 
1 = “Topic addressed in 
Professional 
Development” 

0 = “Topic not 
addressed” OR 

1 = “Topics in which I’d 
like more Professional 
Development” 

0 = “Topic in which I 
would not like more PD” 
Dichotomous 

1 = “Topic addressed in 
Professional 
Development” 

0 = “Topic not 
addressed” OR 

1 = “Topics in which I’d 
like more Professional 
Development” 

0 = “Topic in which I 
would not like more PD” 
Each construct is 
scored as 0 or 1 , 
“Received PD” or “Did 
not Receive PD.” 



Score 1 if any of 
Survey Items m - 
q are checked 
“Topic addressed 
in PD.” OR “Topic 
in which I’d like 
more PD” 

(Range: 0 to 1 ) 



Score 1 if any of 
Survey Items r- u 
are checked 
“Topic addressed 
in PD.” OR “Topic 
in which I’d like 
more PD” 

(Range: 0 to 1 ) 



Score 1 for each 
Survey Item 
checked as 
“Received.” 

Sum Survey Item 



(Range: 0-7) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


Item Scale 


Scoring Metric 


Professional Development (PD) for Teachers (continued) 


Number of PD items 
received in Grouping 


Teacher 
Survey, D5 


h. 


Teacher 
Survey, C5 


h 


Learning styies 


Each construct is 
scored as 0 or 1 , 
“Received PD” or “Did 
not Receive PD.” 


Score 1 for each 
Survey Item 
checked as 
“Received.” 

Sum Survey Item 
scores 

(Range: 0-2) 


i. 


i 


How to organize small group instruction 


Number of PD items 
received in 
Assessment 


Teacher 
Survey, D5 


j- 


Teacher 
Survey, C5 


j- 


How to diagnose reading problems 


Each construct is 
scored as 0 or 1 , 
“Received PD” or “Did 
not Receive PD.” 


Score 1 for each 
Survey Item 
checked as 
“Received.” 

Sum Survey Item 
scores 

(Range: 0-3) 


k. 


k. 


How to administer assessments 


i. 


I. 


How to interpret and use assessment data to 
guide instruction 


Number of PD items 
received in Struggling 
Readers 


Teacher 
Survey, D5 


m. 


Teacher 
Survey, C5 


m. 


How to heip struggling readers with decoding 


Each construct is 
scored as 0 or 1 , 
“Received PD” or “Did 
not Receive PD.” 


Score 1 for each 
Survey Item 
checked as 
“Received.” 

Sum Survey Item 
scores 

(Range: 0-5) 


n. 


n. 


How to heip struggling readers with 
vocabulary 


0. 


0. 


How to help struggling readers with 
comprehension 


p- 


p- 


How to motivate readers 


q- 


q- 


Strategies for teaching reading to students 
with diagnosed learning disabilities 


Number of PD items 
received in 

Organization/Planning 


Teacher 
Survey, D5 


r. 


Teacher 
Survey, C5 


r. 


How to use state/district content standards 
for curriculum planning and teaching. 


Each construct is 
scored as 0 or 1 , 
“Received PD” or “Did 
not Receive PD.” 


Score 1 for each 
Survey Item 
checked as 
“Received.” 

Sum Survey Item 
scores 

(Range: 0-4) 


s. 


s. 


How to align reading curriculum and 
instruction with state/district assessments 


t. 


t. 


How to work with parents 


u. 


u. 


Classroom management 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


Item Scale 


Scoring Metric 


Professional Development (PD) for Building Administrators 


Received PD in 
Teaching Strategies 


Principai 
Survey, F2 


f. 


Principai 
Survey, F1 


f 


How to use the core reading program 


Dichotomous 
1 = “Topic 
addressed in 
Professional 
Development” 
0 = “Topic not 
addressed” 


Score 1 if any of 
Survey Items f - 1 
are checked “Topic 
addressed in PD.” 

(Range: 0 to 1) 


g- 


g- 


How to use children’s literature to teach reading 


h. 


h. 


How to use reading research to guide content of 
instruction 


i. 


i. 


How the core reading program incorporates 
research principals 


j- 


j- 


How to use the supplemental reading program 


k. 


k. 


How to integrate reading and writing instruction 


I. 


i. 


Strategies for teaching reading to ELLs 


Received PD in 
Evaluation 


Principai 
Survey, F2 


m. 


Principai 
Survey, F1 


m. 


How to evaluate a core reading program 


Dichotomous 
1 = “Topic 
addressed in 
Professional 
Development” 
0 = “Topic not 
addressed” 


Score 1 if any of 
Survey Items m - p 
are checked “Topic 
addressed in PD.” 

(Range: 0 to 1) 


n. 


n. 


How to evaluate reading instruction 


0. 


0. 


How to coach teachers in reading instruction 


p- 


p- 


How to manage reading personnel 


Received PD in 
Assessment 


Principai 
Survey, F2 


q- 


Principai 
Survey, F1 


q- 


How to diagnose reading problems 


Dichotomous 
1 = “Topic 
addressed in 
Professional 
Development” 
0 = “Topic not 
addressed” 


Score 1 if any of 
Survey Items q - s 
are checked “Topic 
addressed in PD.” 

(Range: 0 to 1) 


r. 


r. 


How to administer assessments 


s. 


s. 


How to interpret and use assessment data to 
guide instruction 



(Continues) 
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Construct Variabies for Anaiysis of Survey Data (Continued) 



Construct 


2005 Survey 


2007 Survey 


Survey Description 


Item Scale 


Scoring Metric 


Professional Development for Building Administrators (continued) 


Received PD for 
Struggling Readers 


Principal 
Survey, F2 


t. 


Principal 
Survey, F1 


t. 


How to help struggling readers with 
decoding/phonics 


Dichotomous 
1 = “Topic 
addressed in 
Professional 
Development” 
0 = “Topic not 
addressed” 


Score 1 if any of 
Survey Items t - x 
are checked “Topic 
addressed in PD.” 

(Range: 0 to 1) 


u. 


u. 


How to help struggling readers with vocabulary 


V. 


V. 


How to help struggling readers with comprehension 


w. 


w. 


How to motivate readers 


X. 


X. 


Strategies for teaching reading to students with 
diagnosed learning disabilities 


Received PD in 
Organization, 
Management, and 
Support 


Principal 
Survey, F2 


y- 


Principal 
Survey, F1 


y- 


Howto select reading materials 


Dichotomous 
1 = “Topic 
addressed in 
Professional 
Development” 
0 = “Topic not 
addressed” 


Score 1 if any of 
Survey Items y - 
dd are checked 
“Topic addressed in 
PD.” 

(Range: 0 to 1) 


Z. 


Z. 


How to use content standards for curriculum 
planning and teaching 


aa 


aa. 


How to select reading assessments 


bb 


bb. 


Alignment of reading curriculum and instruction with 
state/district assessments 


cc 


CC. 


How to work with parents 


dd 


dd. 


Classroom management 
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